Skip to content

This is the multi-page printable view of this section. .

Return to the regular view of this page.

Pigsty v1.5.1 Documentation

Historical documentation frozen at Pigsty v1.5.1.

Pigsty v1.5.1 Documentation

Battery-Included Open-Source PostgreSQL Distribution

Latest Version: v1.5.1 | Github Repo | Demo

Documentation: EN Docs | 中文文档 | Github Pages

What is Pigsty?

Pigsty is battery-included open-source database distribution, with the latest PostgreSQL kernel, TimescaleDB, PostGIS, Citus & 100+ extensions, along with an entire Infra: Grafana, Prometheus, Loki, Ansible, Docker to support your databases & applications. It also includes common tools for data analysis.

Pigsty is a monitoring & management SRE Solution. Which includes an unparalleled monitoring system with ultimate observability, and high-available PostgreSQL with self-healing architecture. You can deploy various clusters & instances: primary, replica, standby, offline, delayed, cascade, and even Citus, Redis, and Greenplum clusters.

Pigsty is a handy toolbox for developers. It treats Database as Code, Infra as Data. You just describe what database you want, and pigsty will create it for you. You can download, install, deploy, scale, backup, migration with one command. It can be deployed everywhere: a 10k+ core prod env or local 1C/2G VM, cloud, or on-premises.

Pigsty is a secure & thrifty alternative to Cloud RDS/PaaS. It can empower a single DEV/DBA to manage hundreds of databases clusters, with all data under your own control. It can save 50% - 80% cost compared to cloud RDS using ECS or on-premise deployment. And the software itself is completely open-source & free!

Check FEATURES for more detail.

Distribution

Distribution

Observability

Observability

High Availability

High Availability

Architecture

Architecture

Sandbox

Sandbox

Provisioning

Provisioning

User Interface

User Interface

Application

Application

TL; DR

Get a new Linux x86_64 CentOS 7.8 node. with nopass sudo & ssh access, then:

curl -SL https://github.com/Vonng/pigsty/releases/download/v1.5.1/pigsty.tgz | gzip -d | tar -xC ~  # get latest pigsty source
cd ~/pigsty && ./configure                             # pre-check and config templating
./infra.yml                                            # install pigsty on current node

Now you have a battery-included Postgres on port 5432 and infra web services available on port 80.

Check Installation & Demo for details.

Download Packages Directly

Pigsty source & software packages can be downloaded directly via curl in case of no Internet connection:

curl -SL https://github.com/Vonng/pigsty/releases/download/v1.5.1/pkg.tgz -o /tmp/pkg.tgz
curl -SL https://github.com/Vonng/pigsty/releases/download/v1.5.1/pigsty.tgz | gzip -d | tar -xC ~
Mange More Nodes

You can add more nodes to Pigsty with nodes.yml after installing the meta node with infra.yml.

./nodes.yml  -l pg-test      # init 3 nodes of cluster pg-test
Define Postgres Cluster

You can define a HA Postgres Cluster with streaming replication in a few lines of code:

pg-test:
  hosts:
    10.10.10.11: {pg_seq: 1, pg_role: primary}
    10.10.10.12: {pg_seq: 2, pg_role: replica}
    10.10.10.13: {pg_seq: 3, pg_role: replica}
  vars:
    pg_cluster: pg-test

You can create Postgres with different roles by declaring them: primary, replica, standby, delayed, offline, cascade, etc…

Deploy Databases Clusters

You can deploy different types of databases & clusters with corresponding playbooks.

./pgsql.yml         -l pg-test      # init 1-primary & 2-replica pgsql cluster
./redis.yml         -l redis-test   # init redis cluster redis-test
./pgsql-matrixdb.yml -l mx-*         # init MatrixDB cluster mx-mdw,mx-sdw .....

About

Pigsty (/ˈpɪɡˌstaɪ/) is the abbreviation of “PostgreSQL In Graphic STYle.”

Author: Vonng ([email protected])

License: Apache 2.0 License

Beian: 浙ICP备15016890-2号

1 - Introduction

Historical documentation restored from the Pigsty v1.5.1 tag.

Different users have different concerns. If you encounter problems, you are welcome to check the FAQ, submit an Issue, or ask the Community for help.

Beginners

Beginners can visit the Pigsty demo site: http://demo.pigsty.cc for a quick glance.

There are several data applications that are built upon Pigsty to illustrate this distribution: pglog, covid, isd, dbeng, worktime.

You can also deploy SaaS software with docker and get production-grade durability with an external pigsty database.

Developer

Developers are more concerned about the fastest way to download, install and access the database, please refer to Installation.

Pigsty aims at simplicity: you can launch pigsty on fresh CentOS 7.8 nodes with one command without Internet Access.

Pigsty provides Vagrant & Terraform templates for pulling up 4 VMs with one click to deploy a sandbox on a local x86 laptop/PC or cloud. Users can also prepare VMs, cloud VMs, or physical machines for the standard deployment process.

The user can access database service through connstr, and perform essential operation tasks with SOP.

For a deeper understanding of Pigsty’s design and architecture, you can refer to the concept chapter.

Operators

Operators are more concerned with the details of the deployment. The following tutorials provide the details of Pigsty’s installation and deployment.

The tutorial Grafana Backend Database Upgrade shows a complete and representative example of preparing a Postgres cluster exclusively for Grafana.

DBA

DBAs are usually more concerned with the usage of monitoring systems and the specific ways in which they are maintained daily.

DBAs are more concerned with the usage of monitoring systems and the way of daily maintenance.

Monitoring System Tutorial

Daily maintenance management

Professional

For professional users (deep customization, secondary development), Pigsty provides a rich config entry with a customization interface.

2 - Installation

Historical documentation restored from the Pigsty v1.5.1 tag.

It takes 3 steps to install Pigsty: Prepare, Configure, Execute Playbook

There are two typical modes: Singleton & Cluster Management

  • Singleton Meta: Install pigsty on one single meta node. (for PostgreSQL dev/test, RDS monitoring, etc…)
  • Cluster Manage: Initiate control from meta node, and manage & monitoring more nodes & databases.

Singleton Meta

When Pigsty is installed on one single node. It will init pigsty infra with a battery including Postgres as CMDB. You may use that database directly.

TL; DR

Prepare a new node : Linux x86_64 CentOS 7.8.2003, with root or sudo access, then download pigsty

curl -SL https://github.com/Vonng/pigsty/releases/download/v1.5.1/pigsty.tgz | gzip -d | tar -xC ~  # Download Latest Pigsty Source
cd ~/pigsty; ./configure                               # Configure pigsty with ip & template
./infra.yml                                            # Install pigsty on current meta node

If you have available Mac/PC/Laptop/Cloud Account, Sandbox could be handy to create vm for you.

After installation. port 5432 is ready for postgres database connection @ postgres://dbuser_dba:[email protected]:5432/meta.

And a Nginx is serving & proxy all WebUI services @ port 80.

Download Directly via Curl 
curl https://github.com/Vonng/pigsty/releases/download/v1.5.1/pigsty.tgz -o /tmp/pigsty.tgz
curl https://github.com/Vonng/pigsty/releases/download/v1.5.1/pkg.tgz    -o /tmp/pkg.tgz

Access Web Services

Nginx port 80 is the default access endpoint for all Web GUI services. Although it is possible to bypass Nginx add access services directly by port, such as Grafana on 3000, it is highly recommended to use domain names. If you don’t have public domain names, considering use Static DNS on your localhost with make dns.

http://g.pigsty or http://<primary_ip>:3000 is the portal of the monitoring system (username: admin, password: pigsty)

Visit http://<primary_ip>:3000 / http://g.pigsty to visit Pigsty Grafana (username: admin, password: pigsty)

Cluster Manage

Pigsty can also be used as a controller & monitoring infrastructure for production-grade large-scale nodes & databases.

You can initiate control from meta node, and add more nodes into Pigsty.

You can deploy different types of High available PostgreSQL Clusters, Redis clusters, or Greenplum/MatrixDB data warehouse.

./nodes.yml  -l pg-test      # init 3 nodes of cluster pg-test
./pgsql.yml  -l pg-test      # init HA PGSQL Cluster pg-test
./redis.yml  -l redis-test   # init redis cluster redis-test
./pgsql-matrixdb.yml -l mx-*  # init MatrixDB: mx-mdw,mx-sdw

Sandbox

There’s a standard 4-node demo env for pigsty, named ‘Sandbox’. You can create a sandbox on your laptop with Vagrant, or create them on public cloud with Terraform. The same deployment procedure could apply to production deployment.

Take sandbox as an example, install pigsty on meta node 10.10.10.10 with:

./infra.yml     # install pigsty on meta node 10.10.10.10

Nodes Init

Now add 3 more nodes 10.10.10.11, 10.10.10.12, 10.10.10.13 into pigsty nodes.yml

./nodes.yml -l pg-test        # init 3 nodes of cluster pg-test

After that, these 3 nodes are set with DCS service, node exporter & promtail logger. And can be used for subsequent database deployment. check Config: Nodes & Playbook: Nodes for details.

PostgreSQL Deploy

Create a classic 1 primary & 2 replicas HA PostgreSQL cluster with pgsql.yml playbook.

./pgsql.yml -l pg-test  # init pgsql cluster pg-test

Once complete, you can check the newly created PostgreSQL cluster from Monitoring Dashboards.

Check Config: PGSQL, Customize: PGSQL, and Playbook: PGSQL for more details.

Redis Deployment

In addition to the PostgreSQL cluster, you can deploy various other types of clusters and even other types of databases.

For example, to deploy Redis:

./configure -m redis    # use redis config template
./nodes.yml             # init nodes for redis cluster
./redis.yml             # create redis on those nodes

Check Config: REDIS and Playbook: REDIS for more details.

MatrixDB Deployment

To deploy the open-source time-series data warehouse MatrixDB (Greenplum7):

./configure -m mxdb   # Use the sandbox environment MatrixDB configuration file template
./download matrix     # Download the MatrixDB package and build the local source
./infra.yml -e no_cmdb=true # If meta node is part of it, add no_cmdb var to skip CMDB creation
./nodes.yml           # configure all nodes for MatrixDB installation
./pgsql-matrixdb.yml   # Install MatrixDB on those nodes

3 - Features

Historical documentation restored from the Pigsty v1.5.1 tag.

The battery-include, auto-piloting, handy & thrifty distribution for open-source databases.

High Availability / Ultimate Observability / Handy Toolbox / Database as Code / Versatile Scenario / Safety & Thrifty

PostgreSQL Distribution

RedHat for Linux!

  • Pigsty deeply integrates the PostgreSQL 14.4 with powerful extensions: TimescaleDB 2.7, PostGIS 3.2, and Citus 11.0, and hundreds+ of extensions, all Battery-include.

  • Pigsty packs the infrastructure needed for large-scale production environments: Grafana, Prometheus, Loki, Ansible, Consul, Docker, etc. It can also be used as a deployment monitor for other database and application runtimes.

  • Pigsty integrates with common tools for data analysis ecology: Jupyter, ECharts, Grafana, PostgREST, Postgres, which can be used as a data analysis environment, or a low-code data visualization application development platform.

SRE Solution

Auto-Pilot for Postgres! Auto-Pilot! From something to something better for users Use it for fun!

Developer Toolbox

HashiCorp for Database!

  • Pigsty upholds the Infra as Data design philosophy, users can create it in one click using an idempotent playbook with just a few lines of declarative config file describing the database they want. Just like Kubernetes!
  • Pigsty delivers an easy-to-use database toolkit to developers: one-click download installation, automatic configuration; one-click deployment of various open-source databases, one-click migration backup, expansion, and reduction, greatly lowering the threshold of database management use, mass production DBA!
  • Pigsty can simplify database deployment and delivery, solve the problem of unified environment configuration: whether thousands of databases tens of thousands of core production environments, or a local 1C1G laptop can be fully operational; Vagrant-based local sandbox and Terraform-based multi-cloud deployment, cloud on cloud off, pull up with one click!

Open Source RDS

Alternative for RDS!

  • Pigsty can save 50% - 80% of database hardware and software costs compared to cloud vendor RDS with a lower usage threshold and richer features, and junior R&D staff can manage hundreds of databases on their own.

  • Pigsty is modular and can be freely combined and extended on demand. It can deploy and manage various databases in a production environment, or just use them as a host to monitor; it can be used to develop data database visualization demos or support various SaaS applications.

  • Open source, free production-grade database solution to fill in the last missing piece of the cloud-native ecosystem. Stable and reliable, proven over time in large-scale production deployments, with optional professional technical support services.

High Availability

Self-healing & Auto-Piloting.

Taking PostgreSQL as an example, Pigsty creates a database cluster that is distributed and highly available database cluster. As long as any instance of the cluster survives, the cluster can provide complete read-write service and read-only service to the outside world.

Pigsty’s high availability architecture has been tested in production environments. Pigsty uses Patroni + Consul for fault detection, Fencing, automatic failover, and HAProxy, VIP, or DNS for automatic traffic switching, achieving a complete high availability solution at a very low complexity cost, allowing the master-slave architecture of the database to be used with a cloth-like experience. Database-like experience.

The database cluster can automatically perform fault detection and master-slave switching, and common faults can be self-healing within seconds to tens of seconds: RTO < 1min for master failure, read-only traffic is almost unaffected, sync standby cluster RPO = 0 without data loss.

Each database instance in the database cluster is idempotent in use, and any instance can provide full read and write services through the built-in load balancing component HAProxy. Anyone or more Haproxy instances can act as a load balancer for the cluster and distribute traffic through health checks, shielding the cluster members from the outside world. Users can flexibly define services through config and access through various optional methods.

Ultimate Observability

You can’t manage you don’t measure.

Monitoring systems provide metrics on the state of the system and are the cornerstone of operations and maintenance management. [DEMO]

Pigsty comes with a professional-grade monitoring system designed for large-scale database cluster management, based on industry best practices, using Prometheus, Alertmanager, Grafana, and Loki as the monitoring infrastructure. Open source, easy to customize, reusable, portable, no vendor lock-in.

Pigsty is unmatched in PostgreSQL monitoring, presenting about 1200+ categories of metrics through 30+ monitoring panels and thousands of dashboards, covering detailed information from the big global picture to individual objects. Compared with similar products, the coverage of metrics and the richness of monitoring panels are unparalleled, providing irreplaceable value for professional users. The appropriate level of detail is designed to provide an intuitive and convenient management experience for amateur users.

Pigsty’s monitoring system can be used to monitor all kinds of database instances deployed natively: PGSQL, REDIS, GPSQL, etc. It can also be used standalone to monitor existing database instances or remote cloud vendor RDS, or just as a host monitoring, it can also be used as a showcase for data visualization works.

Handy Toolbox

Every additional command line in the install script halves the number of users.

Pigsty takes ease-of-use to the extreme: one command installs and pulls up all components, ready to install in 10 minutes, no dependency on containers and Kubernetes, no Internet access required when using offline packages, and a very low threshold for getting started.

Pigsty has two typical usage models: Standalone and Cluster. It can run completely on local single-core virtual machines and can be used for large-scale production environment database management. Simple operation and maintenance, no worries, no fuss, a one-time solution to all kinds of problems in production environments and personal use of PG.

In standalone mode, Pigsty deploys a complete infrastructure runtime with a single-node PostgreSQL database cluster on that node. For individual users, simple scenarios, and small and micro businesses, you can use this database right out of the box. The single-node model itself is fully functional and self-manageable and comes with a fully-armed and ready-to-use PG database for software development, testing, experiment, demonstration; or data cleansing, analysis, visualization, storage, or direct support for upper-tier applications: Gitlab, Jira, Confluence, UF, Kingdee, Qunhui, etc.

Pigsty has a built-in database management solution with Ansible as the core and is based on this package of command-line tools and graphical interface. It integrates the core functions of database management, including database cluster creation, destruction, expansion and contraction, user, database and service creation, etc.

What’s more, Pigsty packages and provides a complete set of application runtime, which allows users to use the node to manage any number of database clusters. You can initiate control from the node where Pigsty is installed (aka “meta node”) to bring more nodes under Pigsty’s management. You can use it to monitor existing database instances (including cloud vendor RDS) or deploy your own highly available fail-safe PostgreSQL database cluster directly on the node, as well as other kinds of applications or databases, such as Redis and MatrixDB, and Get real-time insights about nodes, databases, and applications.

In addition, Pigsty provides templates for Local Sandbox and Multi-Cloud Deployment based on Vagrant and Terraform, so you can prepare the resources you need for your Pigsty deployment with one click.

Database as Code

A database is a software that manages the data, and a control system is software that manages the database.

Pigsty adopts the design philosophy of Infra as Data, using a declarative configuration similar to Kubernetes, with a large number of optional configuration options to describe the database and the operating environment, and an idempotent preconfigured script to automatically create the required database clusters, providing a private cloud experience.

Pigsty creates the required database clusters from bare metal nodes in minutes based on a list of user config files.

For example, creating a one-master-two-slave database cluster pg-test on three machines requires only a few lines of config and a single command pgsql.yml -l pg-test to create a highly available database cluster as described in the following section.

Example: Customize PGSQL Clusters 
#----------------------------------#
# cluster: pg-meta (on meta node)  #
#----------------------------------#
# pg-meta is the default SINGLE-NODE pgsql cluster deployed on meta node (10.10.10.10)
# if you have multiple n meta nodes, consider deploying pg-meta as n-node cluster too

pg-meta:                                # required, ansible group name , pgsql cluster name. should be unique among environment
  hosts:                                # `<cluster>.hosts` holds instances definition of this cluster
    10.10.10.10:                        # INSTANCE-LEVEL CONFIG: ip address is the key. values are instance level config entries (dict)
      pg_seq: 1                         # required, unique identity parameter (+integer) among pg_cluster
      pg_role: primary                  # required, pg_role is mandatory identity parameter, primary|replica|offline|delayed
      pg_offline_query: true            # instance with `pg_offline_query: true` will take offline traffic (saga, etl,...)
      # some variables can be overwritten on instance level. e.g: pg_upstream, pg_weight, etc...
    #---------------
    # mandatory                         # all configuration above (`ip`, `pg_seq`, `pg_role`) and `pg_cluster` are mandatory
    #---------------
  vars:                                 # `<cluster>.vars` holds CLUSTER LEVEL CONFIG of this pgsql cluster
    pg_cluster: pg-meta                 # required, pgsql cluster name, unique among cluster, used as namespace of cluster resources

    #---------------
    # optional                          # all configuration below are OPTIONAL for a pgsql cluster (Overwrite global default)
    #---------------
    pg_version: 14                      # pgsql version to be installed (use global version if missing)
    node_tune: tiny                     # node optimization profile: {oltp|olap|crit|tiny}, use tiny for vm sandbox
    pg_conf: tiny.yml                   # pgsql template:  {oltp|olap|crit|tiny}, use tiny for sandbox
    patroni_mode: default               # entering patroni pause mode after bootstrap  {default|pause|remove}
    patroni_watchdog_mode: off          # disable patroni watchdog on meta node        {off|require|automatic}
    pg_lc_ctype: en_US.UTF8             # use en_US.UTF8 locale for i18n char support  (required by `pg_trgm`)

    #---------------
    # biz databases                     # Defining Business Databases (Optional)
    #---------------
    pg_databases:                       # define business databases on this cluster, array of database definition
      # define the default `meta` database
      - name: meta                      # required, `name` is the only mandatory field of a database definition
        baseline: cmdb.sql              # optional, database sql baseline path, (relative path among ansible search path, e.g files/)
        # owner: postgres               # optional, database owner, postgres by default
        # template: template1           # optional, which template to use, template1 by default
        # encoding: UTF8                # optional, database encoding, UTF8 by default. (MUST same as template database)
        # locale: C                     # optional, database locale, C by default.  (MUST same as template database)
        # lc_collate: C                 # optional, database collate, C by default. (MUST same as template database)
        # lc_ctype: C                   # optional, database ctype, C by default.   (MUST same as template database)
        # tablespace: pg_default        # optional, default tablespace, 'pg_default' by default.
        # allowconn: true               # optional, allow connection, true by default. false will disable connect at all
        # revokeconn: false             # optional, revoke public connection privilege. false by default. (leave connect with grant option to owner)
        # pgbouncer: true               # optional, add this database to pgbouncer database list? true by default
        comment: pigsty meta database   # optional, comment string for this database
        connlimit: -1                   # optional, database connection limit, default -1 disable limit
        schemas: [pigsty]               # optional, additional schemas to be created, array of schema names
        extensions:                     # optional, additional extensions to be installed: array of schema definition `{name,schema}`
          - { name: adminpack, schema: pg_catalog }    # install adminpack to pg_catalog
          - { name: postgis, schema: public }          # if schema is omitted, extension will be installed according to search_path.
          - { name: timescaledb }                      # some extensions are not relocatable, you can just omit the schema part

      # define an additional database named grafana & prometheus (optional)
      # - { name: grafana,    owner: dbuser_grafana    , revokeconn: true , comment: grafana    primary database }
      # - { name: prometheus, owner: dbuser_prometheus , revokeconn: true , comment: prometheus primary database , extensions: [{ name: timescaledb }]}

    #---------------
    # biz users                         # Defining Business Users (Optional)
    #---------------
    pg_users:                           # define business users/roles on this cluster, array of user definition
      # define admin user for meta database (This user are used for pigsty app deployment by default)
      - name: dbuser_meta               # required, `name` is the only mandatory field of a user definition
        password: md5d3d10d8cad606308bdb180148bf663e1  # md5 salted password of 'DBUser.Meta'
        # optional, plain text and md5 password are both acceptable (prefixed with `md5`)
        login: true                     # optional, can login, true by default  (new biz ROLE should be false)
        superuser: false                # optional, is superuser? false by default
        createdb: false                 # optional, can create database? false by default
        createrole: false               # optional, can create role? false by default
        inherit: true                   # optional, can this role use inherited privileges? true by default
        replication: false              # optional, can this role do replication? false by default
        bypassrls: false                # optional, can this role bypass row level security? false by default
        pgbouncer: true                 # optional, add this user to pgbouncer user-list? false by default (production user should be true explicitly)
        connlimit: -1                   # optional, user connection limit, default -1 disable limit
        expire_in: 3650                 # optional, now + n days when this role is expired (OVERWRITE expire_at)
        expire_at: '2030-12-31'         # optional, YYYY-MM-DD 'timestamp' when this role is expired  (OVERWRITTEN by expire_in)
        comment: pigsty admin user      # optional, comment string for this user/role
        roles: [dbrole_admin]           # optional, belonged roles. default roles are: dbrole_{admin,readonly,readwrite,offline}
        parameters: {}                  # optional, role level parameters with `ALTER ROLE SET`
        # search_path: public         # key value config parameters according to postgresql documentation (e.g: use pigsty as default search_path)
      - {name: dbuser_view , password: DBUser.Viewer  ,pgbouncer: true ,roles: [dbrole_readonly], comment: read-only viewer for meta database}

      # define additional business users for prometheus & grafana (optional)
      - {name: dbuser_grafana    , password: DBUser.Grafana    ,pgbouncer: true ,roles: [dbrole_admin], comment: admin user for grafana database }
      - {name: dbuser_prometheus , password: DBUser.Prometheus ,pgbouncer: true ,roles: [dbrole_admin], comment: admin user for prometheus database , createrole: true }

    #---------------
    # hba rules                                         # Defining extra HBA rules on this cluster (Optional)
    #---------------
    pg_hba_rules_extra:                                 # Extra HBA rules to be installed on this cluster
      - title: reject grafana non-local access          # required, rule title (used as hba description & comment string)
        role: common                                    # required, which roles will be applied? ('common' applies to all roles)
        rules:                                          # required, rule content: array of hba string
          - local   grafana         dbuser_grafana                          md5
          - host    grafana         dbuser_grafana      127.0.0.1/32        md5
          - host    grafana         dbuser_grafana      10.10.10.10/32      md5

    vip_mode: l2                        # setup a level-2 vip for cluster pg-meta
    vip_address: 10.10.10.2             # virtual ip address that binds to primary instance of cluster pg-meta
    vip_cidrmask: 8                     # cidr network mask length
    vip_interface: eth1                 # interface to add virtual ip

In addition, in addition to PostgreSQL, support for Redis deployment and monitoring has been provided since Pigsty v1.3

Example: Redis Cache Cluster 
#----------------------------------#
# redis sentinel example           #
#----------------------------------#
redis-meta:
  hosts:
    10.10.10.10:
      redis_node: 1
      redis_instances:  { 6001 : {} ,6002 : {} , 6003 : {} }
  vars:
    redis_cluster: redis-meta
    redis_mode: sentinel
    redis_max_memory: 128MB

#----------------------------------#
# redis cluster example            #
#----------------------------------#
redis-test:
  hosts:
    10.10.10.11:
      redis_node: 1
      redis_instances: { 6501 : {} ,6502 : {} ,6503 : {} ,6504 : {} ,6505 : {} ,6506 : {} }
    10.10.10.12:
      redis_node: 2
      redis_instances: { 6501 : {} ,6502 : {} ,6503 : {} ,6504 : {} ,6505 : {} ,6506 : {} }
  vars:
    redis_cluster: redis-test           # name of this redis 'cluster'
    redis_mode: cluster                 # standalone,cluster,sentinel
    redis_max_memory: 64MB              # max memory used by each redis instance
    redis_mem_policy: allkeys-lru       # memory eviction policy

#----------------------------------#
# redis standalone example         #
#----------------------------------#
redis-common:
  hosts:
    10.10.10.13:
      redis_node: 1
      redis_instances:
        6501: {}
        6502: { replica_of: '10.10.10.13 6501' }
        6503: { replica_of: '10.10.10.13 6501' }
  vars:
    redis_cluster: redis-common         # name of this redis 'cluster'
    redis_mode: standalone              # standalone,cluster,sentinel
    redis_max_memory: 64MB              # max memory used by each redis instance

Starting with Pigsty v1.4, initial support for MatrixDB (Greenplum7) is provided

Example: MatrixDB Data WareHouse 
#----------------------------------#
# cluster: mx-mdw (gp master)
#----------------------------------#
mx-mdw:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary , nodename: mx-mdw-1 }
  vars:
    gp_role: master          # this cluster is used as greenplum master
    pg_shard: mx             # pgsql sharding name & gpsql deployment name
    pg_cluster: mx-mdw       # this master cluster name is mx-mdw
    pg_databases:
      - { name: matrixmgr , extensions: [ { name: matrixdbts } ] }
      - { name: meta }
    pg_users:
      - { name: meta , password: DBUser.Meta , pgbouncer: true }
      - { name: dbuser_monitor , password: DBUser.Monitor , roles: [ dbrole_readonly ], superuser: true }

    pgbouncer_enabled: true                # enable pgbouncer for greenplum master
    pgbouncer_exporter_enabled: false      # enable pgbouncer_exporter for greenplum master
    pg_exporter_params: 'host=127.0.0.1&sslmode=disable'  # use 127.0.0.1 as local monitor host

#----------------------------------#
# cluster: mx-sdw (gp master)
#----------------------------------#
mx-sdw:
  hosts:
    10.10.10.11:
      nodename: mx-sdw-1        # greenplum segment node
      pg_instances:             # greenplum segment instances
        6000: { pg_cluster: mx-seg1, pg_seq: 1, pg_role: primary , pg_exporter_port: 9633 }
        6001: { pg_cluster: mx-seg2, pg_seq: 2, pg_role: replica , pg_exporter_port: 9634 }
    10.10.10.12:
      nodename: mx-sdw-2
      pg_instances:
        6000: { pg_cluster: mx-seg2, pg_seq: 1, pg_role: primary , pg_exporter_port: 9633  }
        6001: { pg_cluster: mx-seg3, pg_seq: 2, pg_role: replica , pg_exporter_port: 9634  }
    10.10.10.13:
      nodename: mx-sdw-3
      pg_instances:
        6000: { pg_cluster: mx-seg3, pg_seq: 1, pg_role: primary , pg_exporter_port: 9633 }
        6001: { pg_cluster: mx-seg1, pg_seq: 2, pg_role: replica , pg_exporter_port: 9634 }
  vars:
    gp_role: segment               # these are nodes for gp segments
    pg_shard: mx                   # pgsql sharding name & gpsql deployment name
    pg_cluster: mx-sdw             # these segment clusters name is mx-sdw
    pg_preflight_skip: true        # skip preflight check (since pg_seq & pg_role & pg_cluster not exists)
    pg_exporter_config: pg_exporter_basic.yml                             # use basic config to avoid segment server crash
    pg_exporter_params: 'options=-c%20gp_role%3Dutility&sslmode=disable'  # use gp_role = utility to connect to segments

Ubiquitous Deployment

Pigsty can use Vagrant and Virtualbox to pull up and install the required virtual machine environment on your own laptop, or through Terraform, automatically request ECS/VPC resources from your cloud provider, creating and destroying them with a single click.

The virtual machines in the sandbox environment have fixed resource names and IP addresses, making them very suitable for software development testing and experimental demonstrations.

The default sandbox configuration is a single node with 2 cores and 4GB, IP address 10.10.10.10, with a single database instance named pg-meta-1 deployed.

A full version of the sandbox is also available in a four-node version with three additional database nodes, which can be used to fully demonstrate the capabilities of Pigsty’s highly available architecture and monitoring system.

System Requirements
  • Linux kernel, x86_64 processor
  • Use CentOS 7 / RedHat 7 / Oracle Linux 7 or other equivalent operating system distribution
  • CentOS 7.8.2003 x86_64 is highly recommended and has been tested in production environments for a long time

Single Node Basic Specifications

  • Min specification: 1 core, 1GB (OOM prone, at least 2GB of RAM recommended)
  • Recommended specifications: 2 cores, 4GB (sandbox default configuration)
  • A single PostgreSQL instance pg-meta-1 will be deployed
  • In the sandbox, the IP of this node is fixed to 10.10.10.10

Four node basic specifications

  • The meta node requirements are the same as described for a single node

  • Deploy an additional three-node PostgreSQL database cluster pg-test

  • Common database node with min specs: 1 core, 1GB, 2GB RAM recommended.

  • Three nodes with fixed IP addresses: 10.10.10.11, 10.10.10.12, 10.10.10.13

Versatile Scenario

One-click to pull up production SaaS applications, data analysis quickly, low code development visualization large screen

SaaS Software

Pigsty installs Docker by default on the meta node, and you can pull up all kinds of SaaS applications with one click: Gitlab, an open-source private code hosting platform; Discourse, an open-source forum; Mastodon, an open-source social network; Odoo, an open-source ERP software; and UFIDA, Kingdee, and other software.

You can use Docker to pull up stateless parts, modify their database connection strings to use external databases, and get a silky smooth cloud-native management experience with production-grade data persistence. For more details, please refer to Tutorial: Docker Application.

Data Analysis

Pigsty is both a battery-include PostgreSQL distribution and can be used as a data analysis environment, or to make low-code visualization applications. You can go directly from SQL data processing to Echarts plotting in one step, or you can use more elaborate workflows: for example, using PG as the main database, storing data and implementing business logic with SQL; using the built-in PostgREST to automate the back-end API, using the built-in JupyterLab to perform complex data analysis in Python, and using Echarts for data visualization, and Grafana for interaction capabilities.

Pigsty comes with several sample applications for reference.

  • Analysis of PG CSV log samples pglog
  • Visualization of new crown outbreak data covid
  • The global surface weather station data query isd
  • Database prevalence ranking trend dbeng
  • Query the work commuting schedule of a large factory’s worktime

Safety and Thrifty

Pigsty can reduce the total cost of ownership of a database by 50% to 80% and put the data in the hands of the users themselves!

The public cloud database/RDS is a so-called “out-of-the-box” solution, but it delivers a long way from satisfying users: expensive compared to building your own database, many features that require super-user privileges are neutered, stupid UI and pot-luck features, but among all the problems, the most important one is the cloud software safty and cost issues.

Safty

  • Software that runs on your own computer can continue to run even if the software provider goes out of business. But if the company/department providing the cloud software goes out of business or decides to stop supporting it, that software won’t work, and the data you created with that software is locked up. Because the data is only stored in the cloud, not on your own server’s disk, and the only compensation you can expect is usually a chicken scratch voucher.
  • The problem of not being able to customize or scale is further exacerbated in cloud databases. Cloud databases typically do not offer database super users to users, which locks out a whole host of advanced features, as well as the ability to add extensions on your own. In contrast, ‘stream replication’, ‘high availability’, which should be standard in databases, are often sold to users as value-added items.
  • Cloud services may suddenly suspend your account without warning or recourse. You could be judged by an automated system to be in violation of the TOS when you are completely innocent: undocumented use of ports 80 & 53, account blasted and used to send malware or phishing emails, triggering a breach of the TOS. Or hammered over by a cloud vendor for some political reason, such as Parler.
  • The domestic habit of not using SaaS to insist on self-research or open-source is educated by the poor ecological industrial environment for real money. Putting your core asset – data, on someone else’s storage is just like leaving gold over the counter. There is nothing you can do to prevent, monitor, or even be aware of cloud vendors, or simply malicious or curious OPS and DBAs snooping around and stealing your precious data.

Not so with Pigsty, which can be deployed anywhere, including on your own servers. It is open source and free, requires no License, no Internet access, and does not collect any user data. You can run it on your own server until the sea runs out.

Thrifty

The cost of cloud databases is another issue: saving money is an immediate need for users. Public cloud vendors’ RDS may have advantages over traditional commercial databases, but they are still sky-high before building their own open-source databases. According to statistics, the comprehensive holding cost of RDS is up to 2~3x higher than self-build based on cloud servers, and even higher 5~10 times higher than self-build hosted by IDC.

52C/400GB/3TB x 2 Price 5Y Cost/Year
IDC & Your own 810K ¥ 160K ¥
ECS 310K ¥ 63K ¥
RDS 150K ¥ 30K ¥

Pigsty has significant cost advantages over using a cloud database. For example, you can buy the same size cloud server for half the overhead of a cloud database and deploy the database yourself using Pigsty. In this case, you can enjoy most of the ease and convenience of managing a public cloud (IaaS), while instantly saving more than half the overhead.

What’s more, Pigsty can significantly improve user performance: it allows one or two senior DBAs to leave all the trivial chores to the software and easily manage hundreds of database clusters; it also allows a junior R&D staff, after a simple learning training, can quickly reach a senior DBA’s cheap 70% correct level.

Pigsty open source and free, in the premise of providing similar or even exceed the cloud vendor RDS experience, can reduce the comprehensive cost of ownership of the database by 50% ~ 80%, and let the data really control in the hands of the user.

4 - FAQ: Frequently Asked Questions

Historical documentation restored from the Pigsty v1.5.1 tag.

Here are some frequently asked questions. If you have some unlisted questions, Contact Us, or submit an Issue.

Preparation

It would help if you ensured that the machine node hardware specifications and OS meet the installation requirements, see: preparation.

Node Requirement

Warning

At least 1Core/2GB is required from singleton meta installation. An x86_64 Processor is needed. ARM is not supported yet.

Pigsty installation requires at least one node: a minimum specification of 1 core and 2 GB.

If you wish to deploy a self-managed HA PostgreSQL database cluster, it is recommended to use a minimum of 3 nodes with the exact specifications.

OS requirements

Warning

**Pigsty strongly recommends using CentOS 7.8 to avoid meaningless efforts. **

Pigsty development, testing, and packaging are based on CentOS 7.8 by default. CentOS 7.6 is also fully validated. Other CentOS 7. x and its equivalents RHEL7 and Oracle Linux 7 are theoretically OK but have not been tested and verified.

Versioning Policy

Warning

Please use a version-specific Release, not the Github Master branch directly, which may be in an inconsistent state.

Pigsty follows the semantic version numbering rule: <major>. <minor>. <release>.

Major updates imply major fundamental architectural changes, minor version number increases imply package version updates, minor API changes, and other incremental feature changes, and usually include a note on upgrade considerations. The release is usually used for bug fixes and doc updates, and Release version increases do not change package versions (i.e., v1.0.1 and v1.0.0 correspond to the same pkg.tgz).

Pigsty plans to release a Minor Release every 1-3 months and a Major Release every 1-2 years.

Sandbox provisioning

Warning

Use Vagrant to pull up the local sandbox with one click, or use Terraform to create a cloud sandbox at public cloud vendors.

Deploying Pigsty requires physical/VM nodes, and you can provide physical/VMs. However, Pigsty provides IaaS tier resource templates based on Vagrant and HashiCorp to get the VM resources needed to deploy Pigsty 4-node sandbox in one click.

A sandbox is an environment with configuration specifications, object identifiers, IP, and default database predetermined, consisting of a meta node and three nodes, consistent for both local and cloud versions, for development/testing/demo/explanation. Pull up the Vagrant local sandbox using the following command.

make deps    # Install homebrew, and install vagrant and virtualbox via homebrew (requires reboot)
make dns     # Write a static domain name to local /etc/hosts (requires sudo password)
make start   # Pull up a single meta node using Vagrant (4 nodes for start4)

Download

The Pigsty source package is mandatory for installing Pigsty. The offline package is recommended. Please refer to the software download.

Where to download Pigsty source code?

Warning

curl -SL https://github.com/Vonng/pigsty/releases/download/v1.5.1/pigsty.tgz | gzip -d | tar -xC ~

Executing the above command will automatically download the latest stable version of pigsty.tgz and extract it to the ~/pigsty dir. You can also manually download a specific version of Pigsty source code from the following location. If you need to install it in an environment without Internet, you can download it in advance and upload it to the production server via scp/sftp, etc.

https://github.com/Vonng/pigsty/releases/download/v1.5.1/pigsty.tgz   # Github Release
http://download.pigsty.cc/v1.5.1/pigsty.tgz                           # China CDN
https://pan.baidu.com/s/1DZIa9X2jAxx69Zj-aRHoaw?pwd=8su9              # Baidu Yun

Where to download other Pigsty packages?

Warning

./download pigsty pkg app matrix

A download script is provided inside the Pigsty source package for downloading Pigsty-related resources: Pigsty source package: pigsty.tgz / Offline software package: pkg.tgz / MatrixDB/Greenplum package: matrix.tgz / Some SaaS Mirroring and visualization application examples: app.tgz. The source package is mandatory, and the offline package pkg.tgz is recommended ./download pkg will automatically download and extract offline packages.

# download to /tmp/*.tgz
./download pigsty.tgz   # download pigsty source tarball
./download pkg.tgz      # download pigsty offline pkgs
./download app.tgz      # download extra pigsty apps
./download matrix.tgz   # download matrixdb packages
# download and extract
./download pigsty       # download and extract pigsty to ~/pigsty
./download pkg          # download and extract pkg    to /www/pigsty
./download app          # download and extract app    to ~/app
./download matrix       # download and extract matrix to /www/matrix

Where to download Pigsty offline software packages?

Warning

./download pkg or download it automatically when prompted during configuration.

Pigsty’s offline package pkg.tgz packs the required software dependencies.

In ./configure, if the offline package /tmp/pkg.tgz does not exist, the wizard will prompt the user to download it, “Y” will automatically download it from Github or CDN; “N “, the download will be skipped. You can also download the offline package manually from the following location and place it in /tmp/pkg.tgz. Then, it will be used automatically when installing.

curl https://github.com/Vonng/pigsty/releases/download/v1.5.1/pkg.tgz -o /tmp/pkg.tgz
curl http://download.pigsty.cc/v1.5.1/pkg.tgz -o /tmp/pkg.tgz         # China CDN
https://pan.baidu.com/s/1DZIa9X2jAxx69Zj-aRHoaw?pwd=8su9              # Baidu Yun

Download RPMs too slow

Warning

Download from upstream without offline packages, or just remove the problem package and fill in the gaps from available sources

Pigsty offline packages are made based on CentOS 7.8. Otherwise, there may be RPM package dependency issues.

If it is only an isolated RPM dependency issue, you can remove the RPM package in /www/pigsty and delete the marker file /www/pigsty/repo_complete.

And then, when performing the normal installation process, Pigsty will download the missing dependent RPM packages from the upstream or other locally available source specified by repo_upsteram. If there is no available Internet access or local source, please use the same OS environment with a networked node to make an offline package, and then copy it to the production environment for use.

Configuration

Pigsty’s installation, configuration, and deployment are all one-click, with the only configuration being the core of Pigsty.

What is the configured doing?

Warning

Detect the environment, generate the configuration, enable the offline package (optional), and install the essential tool Ansible.

After downloading the Pigsty source package and unpacking it, you need first to execute ./configure to complete the environment configure.

Pigsty will check if the current environment meets the installation requirements and generate the recommended config file pigsty.yml based on the current machine environment. In the files/conf/ directory, there are a series of config files named pigsty-*.yml that can be used as reference templates for configuration in different scenarios, specified by -m.

The Configure installs Ansible, which generally comes with this package as the default source for the node, or from within the offline pkg if it exists.

Where is Pigsty’s config file?

Warning

The source root pigsty.yml is the default, unique config source.

Pigsty has one and only one config file: pigsty.yml in the source root dir, which describes the state of the entire environment.

In ansible.cfg in the same dir: inventory = pigsty.yml specifies this file as the default config file, or you can use the -i parameter when executing the playbook, restricting the use of a config file from another location. In addition, if you use CMDB as the config source, please modify the config in CMDB.

Placeholder IP in the config file

Warning

Pigsty uses 10.10.10.10 as a placeholder for the current node IP, which will be replaced with the primary IP of the current node during the configure.

When the configure detects multiple NICs with multiple IPs on the current node, the config wizard will prompt for the primary IP to be used, i.e., the IP used by the user to access the node from the internal network. Note that please do not use the public IP.

This IP will be used to replace 10.10.10.10 in the config file template.

Does the user need to modify any config?

Warning

Usually, in a singleton installation, there is no need to make any adjustments to the config files.

Pigsty provides 220+ config parameters to customize the entire infra/platform/database. However, there are a few parameters that can be adjusted in advance if needed:

  • When accessing web service components, the domain name is nginx_upstream (some services can only be accessed using the domain name through the Nginx proxy).
  • Pigsty assumes that a /data dir exists to hold all data; you can adjust these paths if the data disk mount point differs from this.

Installation

What was executed during installation?

Warning

When running make install, ansible-playbook is called to perform the preconfigured playbook infra.yml to complete the installation on the meta node.

The configure generates the config file by default and marks the current node as a meta node. And make install executes the Pigsty meta node initialization playbook infra.yml, deploys the infra components, and initializes the meta node like a normal node on which a singleton PostgreSQL is deployed as CMDB.

Downloading RPM packages is too slow

Warning

It is best to use offline packages or configure a proxy server or a local repo.

Pigsty has used domestic yum repos for downloads whenever possible. However, a few packages are still affected by GFW, resulting in slow downloads, such as related software downloaded directly from Github. The following solutions are available.

  1. Pigsty provides an offline package, which pre-packages all software and its dependencies, and can skip the step of downloading software from the Internet.

  2. Specify a proxy server via proxy_env to download via proxy server.

  3. Use other domestic available repos via repo_upsteram.

Remote nodes are not accessible via SSH commands

Warning

Specify a different port via the host instance-level ansible connection parameters.

Consider using Ansible connection parameters if the target machine is hidden behind an SSH springboard machine or if some customizations have been made that cannot be accessed directly using ssh ip. Additional SSH ports can be specified with ansible_port or ansible_host for SSH Alias.

pg-test:
  vars: { pg_cluster: pg-test }
  hosts:
    10.10.10.11: {pg_seq: 1, pg_role: primary, ansible_host: node-1 }
    10.10.10.12: {pg_seq: 2, pg_role: replica, ansible_port: 22223, ansible_user: admin }
    10.10.10.13: {pg_seq: 3, pg_role: offline, ansible_port: 22224 }

Password required for remote node SSH and SUDO

Warning

Use the -k and -K parameters, enter the password at the prompt, and refer to admin provisioning.

When performing deployments and changes, the admin user used must have ssh and sudo privileges for all nodes. Password-free is not required. You can pass in ssh and sudo passwords via the -k|-K parameter when executing the playbook or even use another user to run the playbook via -eansible_host=<another_user>. However, Pigsty strongly recommends configuring SSH passwordless login with passwordless sudo for the admin user.

Sandbox

The Pigsty sandbox provides a standard development/test/demo environment that can be quickly pulled up locally with Vagrant or in the cloud with Terraform.

Vagrant sandbox is too slow to start for the first time

Warning

The first time you use Vagrant to pull up a particular OS repo, it will download the corresponding BOX.

Pigsty sandboxes use CentOS 7 by default, and Vagrant will download the CentOS/7 ISO repo Box the first time the VM is started.

Using a proxy may increase the download speed. Downloading CentOS7 Box only needs to be done the first time the sandbox is started, and will be reused directly when the sandbox is subsequently rebuilt.

Users can also choose to create the required VM manually by downloading the CentOS 7 installation ISO repos.

RPMs error on Aliyun CentOS 7.8 VM

Warning

Aliyun CentOS 7.8 server has DNS caching service nscd installed by default. Just remove it.

Aliyun’s CentOS 7.8 repo has nscd installed by default, locking out the glibc version, which can cause RPM dependency errors during installation.

"Error: Package: nscd-2.17-307.el7.1.x86_64 (@base)"

Run yum remove -y nscd on all nodes to resolve this issue, and with Ansible, you can batch.

ansible all -b -a 'yum remove -y nscd'

Virtual machine time out of sync

Warning

sudo ntpdate -u pool.ntp.org or use make sync4.

The time within the VM may not be consistent with the host after the Virtualbox shutdown. You can try the following command: make sync to force NTP time sync.

sudo ntpdate -u pool.ntp.org
make sync4 # Use NTP POOL
make ss    # Synchronize with Aliyun NTP server

It can solve the problem of no data on the monitoring system after a long hibernation or shutdown and reboot. In addition, restarting the VM can also force a time reset without Internet access: make dw4; make up4.

Why not use containers to hold the database?

Warning

Using Docker/Kubernetes to bloom databases is not mature.

While Docker is good for improving environment compatibility, databases are not among the best scenarios for container use. Docker and Kubernetes have usage thresholds. Pigsty uses bare-metal deployment.

Pigsty was designed initially with a containerized cloud in mind, which is reflected in its declarative implementation of config definitions. It does not require much modification to migrate and transform to a cloud-native solution. It will be refactored using the Kubernetes Operator approach.

Monitoring

How much is the performance storage overhead?

Warning

The monitoring query overhead is negligible, on the order of 100 ms, once every 10 seconds, and a typical instance generates about 2k ~ 5k time series.

Storage depends on the complexity of the user database (workload). For reference: 200 production database instances produce about 16GB of monitoring data in 1 day. Pigsty keeps two weeks of monitoring data by default, which parameters can adjust.

Can you monitor existing PG instances?

Warning

Pigsty does not commit to the quality of monitoring of external instances: PostgreSQL created by Pigsty performs significantly better than hand-crafted instances in most cases.

External databases created by non-Pigsty provisioning modes can be deployed using monly mode.

If Pigsty can manage the instance, consider deploying components such as node_exporter, pg_exporter, and promtail on the target node in the same way as the standard deployment.

Suppose only the URL to access that database (e.g., RDS cloud database instance) is available; in this case, you can use the Monly Deployment mode, where Pigsty monitors the remote PG instance through the pg_exporter instance deployed locally on the meta node.

How do we monitor existing PG instances?

Warning

Why can you still see the removed monitoring objects?

Warning

Use pgsql-remove.yml playbook to remove the monitor target.

INFRA

What components are included in the infra?

Warning

Pigsty provides a complete PaaS environment. Please refer to System Architecture: Infra for more details.

Ansible/Pigsty CLI for launch management and deployment; PostgreSQL on meta nodes as CMDB; Consul Server as meta DB for HA; NTPD and DNS for time and domain name resolution base services; Docker as stateless application deployment base; Prometheus for monitoring metrics, Loki for log collection, Grafana for monitoring/visualization, AlertManager for aggregated alerts, YumRepo for providing local software sources, and Nginx for collecting all WebUI class service access portals externally.

Can you use an existing DCS cluster?

Warning

Pigsty will provide DCS services on a meta node by default, but it is more recommended to use an external cluster of multiple nodes for HA DCS services.

Fill in the dcs_servers with the corresponding cluster, i.e., the external DCS cluster.

There is no correspondence between DCS Server and the meta node: by default, Pigsty installs a single-node Consul Server on the meta node. If the IP of the current node is defined in dcs_servers when performing node initialization, the node is configured with DCS Server services. DCS is used for HA, the primary selection of another database. It is recommended to use a dedicated external DCS cluster of 3 to 5 nodes in a production environment.

NODES

Abort because consul instance already exists

Warning

Pigsty provides a DCS misdeletion protection mechanism, configuring dcs_clean = true.

When the Consul service of the target node already exists, nodes.yml will act on the dcs_clean parameter, and if true, the existing Consul will be erased during initialization.

Pigsty also provides the corresponding SafeGuard parameter: dcs_safeguard.

These parameters can be modified in the config file pigsty.yml or specified at playbook execution time with the extra parameter mechanism.

./nodes.yml -e dcs_clean=true

PGSQL

Abort because Postgres instance already exists

Warning

Pigsty provides a DCS misdeletion protection mechanism, configuring dcs_clean = true.

When the Consul service of the target node already exists, nodes.yml will act on the dcs_clean parameter, and if true, the existing Consul will be erased during initialization.

Pigsty also provides the corresponding SafeGuard parameter: dcs_safeguard.

These parameters can be modified in the config file pigsty.yml or can be specified at playbook execution time with the extra parameter mechanism.

./pgsql.yml -e pg_clean=true

How to ensure HA of PostgreSQL?

Warning

Patroni as HA Agent, Consul as DCS, Haproxy as default traffic distributor, see HA Cluster.

Pigsty uses Patroni to host Postgres, and Patroni uses Consul to reach a consensus on the leader. When the primary fails beyond a threshold (30 seconds), a new election is triggered, and the winner becomes the new cluster primary, with all other replicas following the new cluster primary. The original failed primary is automatically demoted to replica and follows the new primary when it comes online.

The client uses the HAProxy service to access the database, and HAproxy uses an HTTP health check to get the primary-replica role information from Patroni and distribute the traffic accordingly. Pigsty’s database cluster members are idempotent, so as long as anyone instance of the cluster is still alive, read-write and read-only traffic can continue to work and access port 5433 of any one instance. Access to port 5433 of any instance ensures access to the cluster’s central database read and write services.

The availability of DCS is guaranteed by multi-node consensus, so deploying three or more multi-nodes in production environments or using external DCS clusters is recommended.

How to ensure PostgreSQL cluster failure without data loss?

Warning

Use the pg_conf: crit.yml template, or enable sync replication manually.

Crit templates are optimized for data consistency and persistence, with sync commits and data checksums enabled by default. It can ensure no data loss during failover and timely detection of reported silent data corruption due to storage failure, power failure, and other abnormal conditions.

Data corruption causes replica failure

Warning

Find the machine in question, modify the patroni config file clonefrom: false and reload it to take effect.

Pigsty enables the cloneform: true feature by default for all the PGSQL clusters. Suppose an instance cannot complete replica production because of corrupt data files. You can modify the Patroni config file on that instance to set clonefrom to false to avoid pulling data from the evil instance.

5 - Concepts

Historical documentation restored from the Pigsty v1.5.1 tag.

Index

Module Concept Deploy Config Playbook
INFRA Concept: INFRA Deploy: INFRA Config: INFRA Playbook: INFRA
NODES Concept: NODES Deploy: NODES Config: NODES Playbook: NODES
PGSQL Concept: PGSQL Deploy: PGSQL Config: PGSQL Playbook: PGSQL
REDIS Concept: REDIS Deploy: REDIS Config: REDIS Playbook: REDIS

Glossary

Node

Meta

Source Package

Offline Software Package

Sandbox

Production Environment/Prod Env

Singleton Meta

Cluster Manage

Database Cluster

Sandbox

Provisioning

Admin Uesr

Admin Provisioning

Software Provisioning

Configure

Inventory

Config Entry

Cluster

Instance

Service

Sharding Cluster

Playbook

Deploy/Deployment

Primary

Replica

Standby

Offline

Delayed

Hot Standby

Cold Standby

Restore

Alert/Alerting

Metric/Metrics

Dashboard

Panel

Monitor Target

Monly Identity

Identity Parameter

HealthCheck

Service Discovery

Redis Standalone

Redis Native Cluster

Redis Sentinel

6 - Architecture

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty consists of several modules that can be combined according to different scenarios.

Module

Pigsty currently offers four modules.

  • INFRA is Pigsty’s infrastructure module, including monitoring/alerting/visualization/logging/DNS/NTP components.
  • NODES is the node management module for configuring nodes, installing software, and collecting metrics and logs.
  • PGSQL is a PostgreSQL deployment management module, including various PG cluster deployments and monitoring.
  • REDIS is the Redis deployment management module, including Redis standalone/native cluster/sentinel deployment and monitoring.
Module Conecpt Deploy Config Playbook
INFRA Concept: INFRA Deploy: INFRA Config: INFRA Playbook: INFRA
NODES Concept: NODES Deploy: NODES Config: NODES Playbook: NODES
PGSQL Concept: PGSQL Deploy: PGSQL Config: PGSQL Playbook: PGSQL
REDIS Concept: REDIS Deploy: REDIS Config: REDIS Playbook: REDIS

Usage

You can choose which modules to enable on which nodes based on different requirements.

Pigsty will perform a singleton installation by default, initializing the node as a meta node with INFRA, NODES, & PGSQL.

You can join other nodes and install different databases.

Singleton Deploy

If you want to use Pigsty as a battery-included singleton PostgreSQL distribution, install the INFRA, NODES, & PGSQL modules sequentially on a node.

Execute infra.yml playbook to install Pigsty on a singleton meta, deploy infrastructure on the node, and pull up a singleton PostgreSQL database cluster. The node with Pigsty fully installed is called a meta node.

Pigsty is capable of more than that. It can monitor and manage more nodes and databases.

Host Monitoring

If you want to create a large-scale Host Monitoring System, just install the INFRA module on one node and the NODES module on all the monitored nodes. All nodes will be configured with software sources, software packages, DNS, NTP, node monitoring, log collection, and DCS Agent components. Nodes that are included in Pigsty management will come with detailed metrics and can be deployed with various database modules.

Use nodes.yml playbook to install the NODES module on meta nodes for more nodes to be included in Pigsty management.

Database Cluster

After incorporating the nodes into Pigsty, these nodes can be further deployed with various database clusters.

If you want to manage many PostgreSQL clusters, install the PGSQL module again on those nodes included in Pigsty management. You can deploy various PGSQL clusters with one click: singleton, one primary & n replica HA cluster, sync standby cluster, quorum commit cluster, offline replica, standby cluster, delayed replica cluster, Citus cluster, TimescaleDB cluster, MatrixDB data warehouse.

If you want to manage many Redis clusters, install the REDIS module on the included nodes in Pigsty management.

Use pgsql.yml to create a HA PostgreSQL cluster, redis.yml to develop standalone, native, sentinel Redis clusters, and pgsql-matrixdb.yml to deploy the Greenplum/MatrixDB data warehouse.

Pigsty will subsequently add new types of database modules on demand: KAFKA, MINIO, MONGO, etc.

Model

A complete Pigsty system can be called a Deployment or Environment.

For example, production environment, test environment, pre-distribution environment, etc.

A Pigsty deployment consists of two parts: an infra set & multiple clusters, both described by an Inventory.

The cluster contains nodes, instances, and services three core resources: a cluster contains multiple instances deployed on multiple nodes, providing various services. Each database instance will have a more subdivided ER model.

7 - Infrastructure

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty provides a complete, battery-included PaaS infra.

Pigsty, as a battery-included PaaS solution, will integrate the basic infra into the provisioning solution.

Overview

Pigsty deploys a complete set of infra on meta nodes (the currently installed node), including:

Component Port Default Domain Description
Nginx 80 pigsty Web Service Portal
Yum Repo 80 yum.pigsty LocalYum Repo
Grafana 3000 g.pigsty Monitoring System/Visualization Platform
AlertManager 9093 a.pigsty Alert aggregation management component
Prometheus 9090 p.pigsty Monitoring Time Series Database
Loki 3100 l.pigsty Log Collection
Consul 8500 c.pigsty Distributed Configuration Management and Service Discovery
Docker 2375 - Container Platform
PostgreSQL 5432 - Pigsty CMDB
Ansible - - Initiate management commands
Consul DNS 8600 - DNS Service（Optional）
Dnsmasq 53 - DNS Server（Optional）
NTP 123 - NTP Time Server（Optional）

A set of environments contains one or more meta nodes for infra deployment. All infrastructure components are deployed replica-style, except for DCS.

Note

If multiple meta nodes are configured, the DCSs (etcd/consul) on the meta nodes act together as a cluster of DCS servers.

Nginx

Nginx is the portal for all Pigsty’s WebUI services, default port 80.

Many infra components with WebUIs are exposed for service through Nginx, such as Grafana, Prometheus, AlertManager, Consul, and HAProxy traffic management pages. In addition, static file sources such as YumRepo, document, and execution plan visualizers are also served through Nginx.

Nginx forwards access requests to the corresponding upstream component via domain name based on the contents of nginx_upstream. Pigsty strongly recommends using domain names to access the Pigsty UI system for several reasons:

  • Some components only listen to 127.0.0.1 by default.
  • Access via domain name allows you to use Nginx, audit all requests, and facilitate the integration of authentication mechanisms.
  • Domain names are easier to remember and more flexible to configure.

If you do not have an Internet domain name or local DNS resolution, you can add a local static resolution record in /etc/hosts or C:\Windows\System32\drivers\etc\hosts.

Nginx configuration parameters are located at Config: INFRA - NGINX.

Yum Repo

Pigsty will first build a LocalYum repo, which is served by Nginx and is located at /www/pigsty and is available at http://yum.pigsty/pigsty. Pigsty’s offline software package is a compressed Yum Repo dir.

When Pigsty tries to build a local source, if it finds the dir /www/pigsty with the /www/pigsty/repo_complete marker file, it assumes that the local source is built and skips the step of downloading the software from the upstream repo.

The Repo file is located at /www/pigsty.repo and is available by default at http://yum.pigsty/pigsty.repo.

curl http://yum.pigsty/pigsty.repo -o /etc/yum.repos.d/pigsty.repo

You can also use the file local sources without Nginx.

[pigsty-local]
name=Pigsty local $releasever - $basearch
baseurl=file:///www/pigsty/
enabled=1
gpgcheck=0

Yum Repo configuration parameters are located at Config: INFRA - REPO.

Grafana

Grafana is an open-source visualization/monitoring platform, the core of Pigsty WebUI, with default port 3000, which can be accessed via IP:3000 or http://g.pigsty.

Pigsty’s monitoring system is built on Dashboard and connects and jumps via URL. You can quickly locate faults.

In addition, Grafana comes with some visualization plugins, such as the ECharts dashboard.

Grafana configuration parameters are located at Config: INFRA - GRAFANA.

AlertManager

AlertManager is the alerting platform that accompanies Prometheus. The default port is 9093, which can be accessed via IP:9093 or http://a.pigsty.

Prometheus alert events are sent to AlertManager, but if further processing is required, users need to configure it, for example, to provide SMTP service configuration to send alert emails.

Prometheus

Prometheus is the monitoring time-series database, default port 9090, which can be accessed via IP:9090 or http://p.pigsty.

  • Prometheus fetches monitoring objects through the local static file service and associates identity information for them.
  • Prometheus can use Consul service discovery to get monitoring objects automatically.
  • Prometheus pulls monitoring metrics from Exporter, precomputes the processing, and stores them in its TSDB.
  • Prometheus calculates alarm rules and sends the alarm events to Alertmanager for processing.

Prometheus configuration parameters are located at Config: INFRA - PROMETHEUS.

Loki

Loki is the database used for log collection, and Promtail on the node pushes logs to Loki on the meta-node.

Loki configuration parameters are located at Config: INFRA - LOKI.

Consul

Consul Server is used to keep the state of DCS, reach consensus, provide metaDB query service, and also provide DCS-based service discovery.

Consul configuration parameters are located at Config: INFRA - DCS.

Docker

Pigsty installs Docker on the meta-node by default, and you can pull up various stateless applications and use an external database to get production-level persistence.

Docker configuration parameters are located at Config: INFRA - DOCKER.

PostgreSQL

PostgreSQL configuration parameters are located at Config: PGSQL, using CMDB as the configuration source, please refer to the CMDB tutorial.

  • MetaDB for supporting various advanced features (also a standard database cluster, pulled up by Ansible).
  • Ansible for executing playbooks, initiating control, and accessing the CMDB when using dynamic Inventory.
  • Timed task controller (supports backups, cleanups, statistics, patrols, etc.) that accesses the CMDB.

Ansible

By default, Pigsty installs Ansible on the meta-node, a popular operations tool with a declarative configuration style and an idempotent playbook design. The command-line tool pigsty-cli invokes the Ansible Playbook to initiate control.

Ansible configuration parameters are located at Config: INFRA - CONNECT.

Dnsmasq

Dnsmasq provides DNS resolution services (optional)

  • DNS services are optional and can use existing DNS servers.
  • Partial DNS resolution will be forwarded by Consul DNS.

DNSMASQ configuration parameters are located at Config: INFRA - Nameserver.

NTP

NTP service for synchronizing the time of all nodes (optional).

NTP configuration parameters are located at Config: NODES - NTP.

Demo

Pigsty provides a public demo at http://demo.pigsty.cc.

Since the demo is an empty instance of an empty VM with 1 core and 1 GB, please refer to the actual result.

8 - Concept: Nodes

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty use nodes for deployment, nodes cloud be physical machines, VMs, or even Pods.

Pigsty has two types of nodes: meta node and (normal) node.

Meta nodes are used to initiate control, and (normal) nodes are managed under control.

  • Meta node: Run infra.yml playbook to install Pigsty, INFRA, NODES, and PGSQL modules.
  • Node: Run nodes.yml playbook to join Pigsty, and install the NODES module by default.

Meta Node

Meta-Nodes are nodes installed with Pigsty, with admin capability and a complete infra set.

Current node are marked as meta during ./configure, populated in the meta group of inventory.

Pigsty requires at least one meta node per environment. It will be used as a command center for the entire environment. It’s the meta node’s responsibility to keep states, manage configs, launch plays, run tasks, and collect metrics & logs. The infra set is deployed on meta nodes by default: Nginx, Grafana, Prometheus, Alertmanager, NTP, DNS Nameserver, and DCS.

Reuse Meta Node

The meta node can also be reused as a common node, and a PostgreSQL cluster named pg-meta is created by default on the meta. Supporting additional features: CMDB, routine tasks report, extended apps, log analysis & data analysis, etc.

Taking Pigsty Sandbox as an example, the distribution of components on the nodes is shown below.

The sandbox consists of a meta node with 4 nodes. The sandbox is deployed with one set of infra and 2 database clusters. meta is the meta node, deployed with infra and reused as a regular node, deployed with meta DB cluster pg-meta. node-1, node-2, and node-3 are normal nodes deployed with cluster pg-test.

Meta Node Service

The services running on the meta node are shown below.

Component Port Description Default Domain
Nginx 80 Web Service Portal pigsty
Yum 80 LocalYum Repo yum.pigsty
Grafana 3000 Monitoring Dashboards/Visualization Platform g.pigsty
AlertManager 9093 Alert aggregation & notification service a.pigsty
Prometheus 9090 Monitoring Time-Series Database p.pigsty
Loki 3100 Logging Database l.pigsty
Consul (Server) 8500 Distributed Configuration Management and Service Discovery c.pigsty
Docker 2375 Container Platform -
PostgreSQL 5432 Pigsty CMDB -
lAnsible - Controller -
Consul DNS 8600 DNS Service Discovery powered by Consul -
Dnsmasq 53 DNS Name Server（Optional） -
NTP 123 NTP Time Server（Optional） -
Pgbouncer 6432 Pgbouncer Connection Pooling Service -
Patroni 8008 Patroni HA Component -
Haproxy Primary 5433 Primary Pooling: Read/Write Service -
Haproxy Replica 5434 Replica Pooling: Read-Only Service -
Haproxy Default 5436 Primary Direct Connect Service -
Haproxy Offline 5438 Offline Direct Connect: Offline Read Service -
Haproxy Admin 9101 HAProxy admin & metrics -
PG Exporter 9630 PG Monitoring Metrics Exporter -
PGBouncer Exporter 9631 PGBouncer Monitoring Metrics Exporter -
Node Exporter 9100 Node monitoring metrics -
Promtail 9080 Logger agent -
vip-manager - Bind VIP to the primary

Meta Node & DCS

By default, DCS Servers (Consul or Etcd) will be deployed on the meta nodes, or you can use External DCS Cluster. Any infra outside DCS will be deployed on the meta node as a peer-to-peer copy. The number of meta nodes requires a minimum of 1, recommends 3, and recommends no more than 5.

Warning

DCS Servers are used for leader election in HA Scenarios. Shutting down the DCS servers will demote ALL clusters, which reject any writes by default! So make sure you have enough availability on these DCS Servers, at least stronger than PostgreSQL itself. It’s recommended to add more meta nodes or use an external independently maintained, HA DCS cluster for production-grade deployment.

Multiple Meta Nodes

Usually, one meta node is sufficient for basic usage, two meta nodes can be used as standby backup, and 3 meta nodes can support a minimal meaningful production-grade DCS Servers themselves!

Pigsty will set DCS Servers on all meta nodes by default for the sake of “Battery-Included”. But it’s meaningless to have more than 3 meta nodes. If you are seeking HA DCS Servies. Using an external DCS Cluster with 3~5 nodes would be more appropriate.

Meta nodes are configured under all.children.meta.host in the inventory. They will be marked with meta_node: true flag. The node runs configure will be marked as meta, and multiple meta nodes have to be configured manually, check pigsty-dcs3.yml for example.

If you are not using any external DCS as an arbiter. It requires at least 3 nodes to form a meaningful HA Cluster that allows one node failure.

Node

You can manage more nodes with Pigsty, and use them to deploy various databases or your applications.

The nodes managed by Pigsty are adjusted by nodes.yml to the state described by Config: NODES, and the node monitoring and log collection components are installed so you can check the node status and logs from the monitoring system.

Node Identity

Each node has identity parameters that are configured by parameters in <cluster>.hosts and <cluster>.vars.

There are two important node identity parameters: nodename and node_cluster, which will be used as the node’s instance identity (ins) and cluster identity (cls) in the monitoring system. nodename and node_cluster are NOT REQUIRED since they all have proper default values: Hostname and constant nodes.

Besides, Pigsty uses an IP address as a unique node identifier, too. Which is the inventory_hostname reflected as the key in the <cluster>.hosts object. A node may have multiple interfaces & IP addresses. But you must explicitly designate one as the PRIMARY IP ADDRESS. Which should be an intranet IP for service access. It’s not mandatory to use that same IP address to ssh from the meta node, you can use ssh tunnel & jump server with Ansible Connect parameters.

Name Type Level Attribute Description
inventory_hostname ip - REQUIRED Node IP
nodename string I Optional Node Name
node_cluster string C Optional Node Cluster Name

The following cluster configuration declares a three-node cluster.

node-test:
  hosts:
    10.10.10.11: { nodename: node-test-1 }
    10.10.10.12: { pg_hostname: true } # Borrowed identity pg-test-2
    10.10.10.13: {  } # Use the original hostname: node-3
  vars:
    node_cluster: node-test
host node_cluster nodename instance
10.10.10.11 node-test node-test-1 pg-test-1
10.10.10.12 node-test pg-test-2 pg-test-2
10.10.10.13 node-test node-3 pg-test-3

IIn the monitoring system, the time-series monitoring data are labeled as follows.

node_load1{cls="pg-meta", ins="pg-meta-1", ip="10.10.10.10", job="nodes"}
node_load1{cls="pg-test", ins="pg-test-1", ip="10.10.10.11", job="nodes"}
node_load1{cls="pg-test", ins="pg-test-2", ip="10.10.10.12", job="nodes"}
node_load1{cls="pg-test", ins="pg-test-3", ip="10.10.10.13", job="nodes"}

Node Services

Component Port Description
Consul Agent 8500 Distributed Configuration Management and Service Discovery
Node Exporter 9100 Node Monitoring Metrics Exporter
Promtail 9080 Collection of Postgres, Pgbouncer, Patroni logs (Optional)
Consul DNS 8600 DNS Service

PGSQL Node

A PGSQL Node is a node with a PGSQL module installed.

Pigsty uses exclusively deploy policy for PGSQL. This means the node’s identity and pgsql’s identity are exchangeable. The pg_hostname parameter is designed to assign the Postgres identity to its underlying node: pg_instance and pg_cluster will be assigned to the node’s nodename & node_cluster.

In addition to node default services, the following services are available on PGSQL nodes.

Component Port Description
Postgres 5432 Pigsty CMDB
Pgbouncer 6432 Pgbouncer Connection Pooling Service
Patroni 8008 Patroni HA Component
Consul 8500 Distributed Configuration Management and Service Discovery
Haproxy Primary 5433 Primary connection pool: Read/Write Service
Haproxy Replica 5434 Replica connection pool: Read-only Service
Haproxy Default 5436 Primary Direct Connect Service
Haproxy Offline 5438 Offline Direct Connect: Offline Read Service
Haproxy service 543x Customized Services
Haproxy Admin 9101 Monitoring metrics and traffic management
PG Exporter 9630 PG Monitoring Metrics Exporter
PGBouncer Exporter 9631 PGBouncer Monitoring Metrics Exporter
Node Exporter 9100 Node Monitoring Metrics Exporter
Promtail 9080 Collection of Postgres, Pgbouncer, Patroni logs (Optional)
Consul DNS 8600 DNS Service
vip-manager - Bind VIP to the primary

Node Interaction

Here’s an example of interactions between a meta node & a common node.

The interaction between the meta node and common nodes are:

  • Database cluster/node domain name resolved by Nameserver on meta node. (optional)
  • Database node software installation will use Yum Repo on meta.
  • Prometheus collects database cluster/node monitoring metrics on meta.
  • Database logs are collected by Promtail and sent to Loki.
  • Pigsty will control database nodes from the meta node:
    • Cluster creation, scale in / scale out, instance/cluster recycling
    • Creating business users & databases, modifying services routes & HBA rules.
    • Log collection, vacuum analyze, backup, and other routine tasks, etc.
  • Node’s Consul will sync locally registered services to the DCS Servers.
  • The database node will synchronize time from the meta node (or other NTP server).

9 - Concept: PGSQL

Historical documentation restored from the Pigsty v1.5.1 tag.

This article introduces the core concepts required for PostgreSQL cluster management.

PGSQL Cluster

PGSQL for production environments is organized in clusters, which clusters are logical entities consisting of a set of database instances associated by primary-replica. Each database cluster is a self-organizing business service unit consisting of at least one database instance.

Sandbox

Clusters are the basic business service units, and the following diagram shows the replication topology in a sandbox where pg-meta-1 constitutes a database cluster pg-meta. In contrast, pg-test-1, pg-test-2, and pg-test-3 form another logical cluster pg-test.

pg-meta-1
(primary)

pg-test-1 -------------> pg-test-2
(primary)      |         (replica)
               |
               ^-------> pg-test-3
                         (replica)

High-Availability

Primary Failure RTO ≈ 30s~1min, RPO < 10MB, Replica Failure RTO≈0 (reset current conn)

Pigsty creates a HA PostgreSQL cluster by default. Pigsty can automatic failover, and read-only business traffic is not affected; the impact of reading and write traffic depends on the specific configuration and load, usually in a few seconds to tens of seconds.

By default, Pigsty deploys clusters in availability first mode. When the primary goes down, data not replicated to the replica part may be lost (generally about a few hundred KB, no more than 10 MB); you can refer to Sync Standby and use consistency first mode, RPO = 0 in this mode.

Pigsty’s HA is achieved using Patroni + HAProxy, with the former failing over and the latter switching over traffic.

Patroni uses DCS service for heartbeat preservation, and the primary will register a 15-second lease by default and renew it periodically. When the primary fails to renew the lease, the lease is released, and a new primary election round is triggered. Usually, the one with the lowest delay is elected as the new primary. The cluster enters a new timeline, and all other clusters, including the old primary, re-follow the new primary.

HAProxy automatically detects the state of the instances and distributes the traffic correctly. Haproxy is stateless and deployed uniformly on each node/instance. All HAProxy can act as service access for the cluster. For example, the Primary service on port 5433 will use HTTP GET ip:8008/primary health check to get information from all Patroni in the cluster, find out the primary, and distribute traffic to the primary.

Interaction

On a singleton node/instance, the components work with each other through the following connections.

  • vip-manager gets the primary information by querying the Consul and binds the cluster-specific L2 VIP to the primary (default sandbox access).
  • Haproxy is the database traffic portal for exposing services, with different ports (543x) distinguishing between different services.
    • Haproxy port 9101 exposes Haproxy monitoring metrics and provides Admin interface traffic control.
    • Haproxy port 5433 defaults point to primary connection pool port 6432
    • Haproxy port 5434 defaults point to replica connection pool port 6432
    • Haproxy port 5436 defaults point to primary 5432 port.
    • Haproxy port 5438 defaults point to offline 5432 port.
  • Pgbouncer is used for pooling database connections, buffering failures, and exposing additional metrics.
    • Production services (HF non-interactive, 5433/5434) must be accessed via Pgbouncer.
    • Directly connected services (management and ETL, 5436/5438) must be directly connected, bypassing Pgbouncer.
  • Postgres provides database services that form a primary-replica cluster via streaming replication.
  • Patroni oversees the Postgres service, primary-replica election and switchover, health checks, and config management.
    • Patroni uses Consul to reach Consensus, the basis for the primary election.
  • The Consul Agent is used to issue configs, accept service registrations, service discovery, and provide DNS queries.
    • All services that use the port are registered with Consul.
  • PGB Exporter, PG Exporter, and Node Exporter are used to expose database, connection pool, and node monitoring metrics.
  • Promtail is the log collection component used to send the collected PG, PGB, Patroni, and node logs to the infrastructure Loki.

ER Model

In Pigsty, PostgreSQL has four types of core entities.

Entities

  • Cluster is the basic autonomous unit, uniquely identified by user designation, expressing business meaning, and serving as a top-level namespace.
  • The clusters contain a series of Nodes at the hardware level, i.e., physical machines and VMs (or Pods) that IP can uniquely identify.
  • The cluster contains a series of Instances at the software level, i.e., software servers, which can be uniquely identified by IP: Port.
  • The cluster contains a series of Services at the service level, i.e., accessible domains and ports that can be uniquely identified by domains.

Naming Pattern

  • Cluster naming can use any name that satisfies the DNS domain name specification, not with a dot ( [a-zA-Z0-9-]+).
  • Node naming uses the cluster name as a prefix, followed by -and an ordinal integer number.
  • Instance naming can be consistent with the node naming, i.e., ${cluster}-${seq}.
  • Service naming also uses the cluster name as the prefix, followed by - to connect the service specifics, such as primary, replica, offline, delayed, etc.

Take the test database cluster pg-test for a sandbox as an example.

  • One cluster: The database cluster for testing is named pg-test.
  • Two roles: primary and replica.
  • Three instances: The cluster consists of three database instances: pg-test-1, pg-test-2, pg-test-3.
  • Three nodes: The cluster is deployed on three nodes: 10.10.10.11, 10.10.10.12, and 10.10.10.13.
  • Four services:

Identity Parameter

Entities and identities are a conceptual model, and the following describes the implementation in Pigsty.

pg_cluster，pg_role， and pg_seq are identity parameters used to generate entity identities.

In addition to the IP address, these three parameters are the minimum set of parameters necessary to define database clusters.

  • Cluster Identity：pg_cluster ： {{ pg_cluster }}
  • Instance Identity：pg_instance ： {{ pg_cluster }}-{{ pg_seq }}
  • Service Identity：pg_service ：{{ pg_cluster }}-{{ pg_role }}
  • Node Identity：nodename：
    • pg_hostname: true: Use the same as pg_instance：{{ pg_cluster }}-{{ pg_seq }}
    • pg_hostname: false: Explicitly specifying {{ nodename }} is used directly. Otherwise, the existing hostname is used.

The following is a sample definition of a pg-test cluster in a sandbox.

pg-test:
  hosts:
    10.10.10.11: {pg_seq: 1, pg_role: replica}
    10.10.10.12: {pg_seq: 2, pg_role: primary}
    10.10.10.13: {pg_seq: 3, pg_role: replica}
  vars:
    pg_cluster: pg-test
    pg_hostname: true     # The identity of the PG instance as the identity of the node(1:1)

The three members of the cluster are identified as follows.

host cluster instance service nodename
10.10.10.11 pg-test pg-test-1 pg-test-primary pg-test-1
10.10.10.12 pg-test pg-test-2 pg-test-replica pg-test-2
10.10.10.13 pg-test pg-test-3 pg-test-replica pg-test-3

In the monitoring system, the monitoring time series data is labeled as：

pg_up{cls="pg-meta", ins="pg-meta-1", ip="10.10.10.10", job="pgsql"}
pg_up{cls="pg-test", ins="pg-test-1", ip="10.10.10.11", job="pgsql"}
pg_up{cls="pg-test", ins="pg-test-2", ip="10.10.10.12", job="pgsql"}
pg_up{cls="pg-test", ins="pg-test-3", ip="10.10.10.13", job="pgsql"}

Cluster

A cluster is the basic autonomous business unit, which means that the cluster can provide services as a whole. Note that cluster here is a software-level concept, not to be confused with PG Cluster (database set cluster, i.e., a data directory containing multiple PGs of a singleton) or Node Cluster (machine cluster).

A cluster is one of the basic management units, and an organizational unit is used to unify various sources. A PG cluster may include.

  • Three physical machine nodes
  • One primary instance provides database read and writes services to.
  • Two replica instances provide read-only copies of the database.
  • Two exposed services: read-write service, and read-only copy service.

Cluster Naming Pattern

Each cluster has a unique identity. In this case, a database cluster named pg-test is defined.

The cluster name is similar to the role of a namespace. All sources belonging to this cluster will use this namespace.

The cluster identity (cls) must be unique within a set of environments, and naming patterns that conform to the DNS standard RFC1034 is recommended.

A good cluster name should use only lowercase letters, numbers, and the hyphen -and use letter starters.

cluster_name := [a-z][a-z0-9-]*

Cluster naming should not include the dot. A popular naming pattern uses dot-separated hierarchical identities, such as com.foo.bar. This naming is simple, but the number of domain hierarchies is not controllable. The most intuitive example is Pods in Kubernetes, where Pod naming patterns do not allow.

Connotation of cluster naming is recommended by-separated two-paragraph and three-paragraph names.

<cluster type>-<business>-<business line>

Typical cluster names include: pg-meta, pg-test-fin, pg-infrastructure-biz.

Instance

An instance refers to a specific database server, which can be a single process, a group of processes, or several associated containers within a Pod. The critical elements of an instance are.

  • Can be uniquely identified by the instance identity (ins).
  • Can handle requests (regardless of whether the request is received from a database, a connection pool, or a load balancer).

Instance Naming Pattern

Instances belong to clusters, and each instance has its unique identity within the cluster. The instance identity ins is recommended to use a naming pattern consistent with Kubernetes Pods: i.e., cluster name linked to an ordinal integer number in increments from 0/1 <cls>-<seq>.

Pigsty names the database instances in a cluster by default, increasing order starting from 1. For example, the database cluster pg-test has three database instances: pg-test-1, pg-test-2, and pg-test-3.

Once the instance name ins is assigned immutable, the instance will be used for the entire lifetime of the cluster.

In addition, with a singleton deployment, the database instance and the machine node can use each other’s identities.

Node

A Node is an abstraction of a hardware resource, usually referring to a working machine, whether a physical machine (bare metal), a VM or a Pod in Kubernetes.

Note

Note that Node in Kubernetes is an abstraction of hardware sources, but in reality, the concept of Node is similar to the concept of Pod in Kubernetes.

The key features of a Node are.

  • Nodes are abstractions of hardware sources that can run software services and deploy database instances.
  • Nodes can use IP as unique identities.

Node Naming Pattern

Pigsty uses ip as the node’s unique identity. If the machine has more than one IP, the actual access IP specified in the inventory will prevail. The hostname nodename, database instance identity ins, and node identity ip correspond to each other in Pigsty and can be cross-used as identities for database instances, machine nodes, and HAProxy load balancers.

The node naming is consistent with the database instance and remains the same throughout the cluster’s life.

Service

A service is a named abstraction of a software service (e.g., Postgres, Redis). Services have various implementations, but the key elements are:

  • An addressable and accessible service name for providing access:
    • A DNS domain name (pg-test-primary)
    • An Nginx/Haproxy Port
  • Service traffic routing and load balancing mechanism for deciding which instance handles requests:
    • DNS L7: DNS resolution records
    • HTTP Proxy: Nginx/Ingress L7: Nginx Upstream Config
    • TCP Proxy: Haproxy L4: Haproxy Backend Config
    • Kubernetes: Ingress: Pod Selector.
    • The service also needs to decide which component will handle the request: the connection pool, or the database itself.

For more information about services, see the chapter Services.

Service Naming Pattern

The service identity (svc) consists of cls as a namespace and (role) as the service bearer.

In a PostgreSQL cluster, instances have different identities: primary, replica, standby, offline, and delayed. Different instances will provide different services; direct connection to the database and access to the database through connection pools are services of varying nature. It is common to use the role of the service target to identify the service, e.g., in the database cluster pg-test.

  • A service that points to the primary connection pool (primary) role instance is called pg-test-primary.
  • A service that points to a replica connection pool (replica) role is called pg-test-replica.
  • A service that points to an (offline) is called pg-test-offline.
  • A service that points to a (standby) is called pg-test-standby.

Note that services are not enough to divide pairs of instances. The same service can point to multiple instances. However, the same instance can also handle requests from different services.

10 - Concept: Redis

Historical documentation restored from the Pigsty v1.5.1 tag.

This article introduces the core concepts required for Redis cluster management.

Deploy: Redis ｜Config: Redis | Playbook: Redis

ER Model

The Redis entity concept model is almost identical to PostgreSQL and includes the Cluster and Instance. Note that Cluster here does not refer to the clusters in Redis’ native clusters.

The core difference is that Redis typically uses multiple singleton instances, with multiple Redis instances typically deployed on a single physical/VM to take advantage of multi-core CPUs.

In Pigsty-managed Redis, it is not yet possible to deploy two Redis instances from different clusters on a node, but this does not affect the deployment of multiple independent Redis instances on a node.

Redis Identity

The identity parameters are the information that must be provided when defining a Redis cluster.

Name Attribute Description Example
redis_cluster MUST, cluster level cluster name redis-test
redis_node MUST，node level Node Number 1,2
redis_instances MUST，node level Instance Definition { 6001 : {} ,6002 : {}}
  • redis_cluster: Identifies the Redis cluster name, configured at the cluster level, as the top-level namespace for cluster sources.
  • redis_node: Identifies the number of the node in the cluster.
  • redis_instances: A JSON object with the Key as the instance port and the Value as a JSON object containing the instance-specific configuration.

11 - PGSQL Service & Access

Historical documentation restored from the Pigsty v1.5.1 tag.

Defining PostgreSQL service and achieving stable, reliable, and HA access through LB and connection pooling.

The concepts of Service and Access are proposed for HA PostgreSQL clusters.

Personal User

After completing the singleton deployment, port 5432 of this node provides PostgreSQL services, and port 80 provides UI class services.

On the current meta node, executing psql with no parameters using the admin user can connect directly to the local pre-defined meta database.

When accessing PG from the host using the client tool, you can use the URL.

psql postgres://dbuser_dba:[email protected]/meta         # dbsu direct connection
psql postgres://dbuser_meta:[email protected]/meta       # business user direct connect

You can use the admin user specified by pg_admin_username and pg_admin_password or a business user (dbuser_meta) pre-defined in the meta database to access this database.

When using a HA database cluster deployed with Pigsty, it is not recommended to access the database service using IP direct connection.

Service

Service in the form of functionality that a database cluster provides.

In a production env, a replication-based primary-replica database cluster is used. There is one and only one primary in the cluster that can accept writes, while the other replicas will continuously get logs from the primary to keep up with it. Also, replicas can host read-only requests.

In addition, for production envs with short high-frequency connections, we also pool requests via Pgbouncer to reduce connection creation overhead. However, in ETL and change execution scenarios, we need to bypass the connection pool and access the database directly.

In addition, HA clusters have a failover feature that causes changes to the cluster’s primary. HA clustering solutions, therefore, require that write traffic can automatically adapt to changes in the cluster’s primary.

These different access requirements (read/write separation, pooling, and direct connection, failover auto-adaptation) are eventually abstracted into the concept of Service.

In general, a database cluster must provide a service.

  • read-write service (primary): can write to the database

For a production database cluster, at least two services should be provided.

  • read-write service (primary): can write to the database

  • read-only service (replica): access to the replica

There may be other services.

  • offline: For ETL and personal queries.
  • standby: Read-only service with synchronous commit and no replication delay.
  • delayed: Allows to access old data before a fixed time interval.
  • default: Service that allows admin users to manage the database directly, bypassing the connection pool.

Default Services

Pigsty provides four services by default: primary, replica, default, and offline.

New services can be defined for global or individual clusters via config files.

service port purpose description
primary 5433 production read/write connect to primary via connection pool
replica 5434 production read-only connection to replica via connection pool
default 5436 management direct connection to primary
offline 5438 ETL/personal user direct connection to offline

Take the meta DB pg-meta as an example:

psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5433/meta   # production read/write
psql postgres://dbuser_meta:DBUser.Meta@pg-meta:5434/meta   # production read-only
psql postgres://dbuser_dba:DBUser.DBA@pg-meta:5436/meta     # Direct connect primary
psql postgres://dbuser_stats:DBUser.Stats@pg-meta:5438/meta # Direct connect offline

These four services are described in detail below.

Primary Service

The Primary service is used for online production read and write access, and it maps the cluster’s port 5433 to the primary connection pool (default 6432) port.

The Primary service selects all instances in the cluster as members, but only the primary can take on traffic because there is one and only one instance /primary with a true health check.

# primary service will route {ip|name}:5433 to primary pgbouncer (5433->6432 rw)
- name: primary           # service name {{ pg_cluster }}-primary
  src_ip: "*"
  src_port: 5433
  dst_port: pgbouncer     # 5433 route to pgbouncer
  check_url: /primary     # primary health check, success when instance is primary
  selector: "[]"            # select all instance as primary service candidate

The HA component Patroni on the primary returns 200 against the Primary health check and is used to ensure that the cluster does not have another primary.

When the cluster fails over, the health check is true for the new primary and false for the old one, so traffic is migrated to the new primary. The business side will notice about 30 seconds of Primary service unavailability time.

Replica Service

The Replica service is used to online produce read-only access, and it maps the cluster’s port 5434, to the replica connection pool (default 6432) port.

The Replica service selects all instances in the cluster as members, but only those with an accurate health check /read-only can take on traffic, and that health check returns success for all instances that can take on read-only traffic.

By default, only replicas carry read-only requests, and the Replica service defines selector_backup, a selector that adds the cluster’s primary as a backup instance to the Replica service. The primary will only start taking read-only traffic when all replicas are down.

Another role as a backup instance is offline, which is usually dedicated to OLAP/ETL/personal queries and is not suitable for mixing with online queries, so offline is only used to take on read-only traffic when all replicas are down.

# replica service will route {ip|name}:5434 to replica pgbouncer (5434->6432 ro)
- name: replica           # service name {{ pg_cluster }}-replica
  src_ip: "*"
  src_port: 5434
  dst_port: pgbouncer
  check_url: /read-only   # read-only health check. (including primary)
  selector: "[]"          # select all instance as replica service candidate
  selector_backup: "[? pg_role == `primary` || pg_role == `offline` ]"

Default Service

The Default service is used for online primary direct connections, which map the cluster’s port 5436 to the primary Postgres (default 5432) port.

Default service targets interactive read and writes access, including executing admin commands, performing DDL changes, connecting to the primary to perform DML, and performing CDC. Default service forwards traffic directly to Postgres, bypassing Pgbouncer.

The Default service is similar to the Primary service, using the same config entry.

# default service will route {ip|name}:5436 to primary postgres (5436->5432 primary)
- name: default           # service's actual name is {{ pg_cluster }}-default
  src_ip: "*"             # service bind ip address, * for all, vip for cluster virtual ip address
  src_port: 5436          # bind port, mandatory
  dst_port: postgres      # target port: postgres|pgbouncer|port_number , pgbouncer(6432) by default
  check_method: http      # health check method: only http is available for now
  check_port: patroni     # health check port:  patroni|pg_exporter|port_number , patroni by default
  check_url: /primary     # health check url path, / as default
  check_code: 200         # health check http code, 200 as default
  selector: "[]"          # instance selector
  haproxy:                # haproxy specific fields
    maxconn: 3000         # default front-end connection
    balance: roundrobin   # load balance algorithm (roundrobin by default)
    default_server_options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

Offline Service

Offline service is used for offline access and personal queries. It maps the cluster’s 5438 port, to the offline Postgres (default 5432) port.

The Offline service targets interactive read-only access, including ETL, offline analytics queries, and individual user queries. Offline service also forwards traffic directly to Postgres, bypassing Pgbouncer.

Offline instances are those where pg_role is offline or tagged with pg_offline_query. The other replica outside the Offline will act as a backup instance for Offline and will still be able to get services from other replicas when the Offline is down.

# offline service will route {ip|name}:5438 to offline postgres (5438->5432 offline)
- name: offline           # service name {{ pg_cluster }}-offline
  src_ip: "*"
  src_port: 5438
  dst_port: postgres
  check_url: /replica     # offline MUST be a replica
  selector: "[? pg_role == `offline` || pg_offline_query ]"         # instances with pg_role == 'offline' or instance marked with 'pg_offline_query == true'
  selector_backup: "[? pg_role == `replica` && !pg_offline_query]"  # replica are used as backup server in offline service

User-Defined Service

In addition to the default services configured by pg_services above, users can define additional services for the PostgreSQL cluster in the pg_services_extra config entry.

A cluster can define multiple services, each containing any number of cluster members, distinguished by port. The following code defines a new service standby that uses port 5435 to provide sync read functionality. This service will read from standby (or primary) in the cluster, thus ensuring that all reads are done without latency.

# standby service will route {ip|name}:5435 to sync replica's pgbouncer (5435->6432 standby)
- name: standby                   # required, service name, the actual svc name will be prefixed with `pg_cluster`, e.g: pg-meta-standby
  src_ip: "*"                     # required, service bind ip address, `*` for all ip, `vip` for cluster `vip_address`
  src_port: 5435                  # required, service exposed port (work as kubernetes service node port mode)
  dst_port: postgres              # optional, destination port, postgres|pgbouncer|<port_number>   , pgbouncer(6432) by default
  check_method: http              # optional, health check method: http is the only available method for now
  check_port: patroni             # optional, health check port: patroni|pg_exporter|<port_number> , patroni(8008) by default
  check_url: /read-only?lag=0     # optional, health check url path, / by default
  check_code: 200                 # optional, health check expected http code, 200 by default
  selector: "[]"                  # required, JMESPath to filter inventory ()
  selector_backup: "[? pg_role == `primary`]"  # primary used as backup server for standby service (will not work because /sync for )
  haproxy:                        # optional, adhoc parameters for haproxy service provider (vip_l4 is another service provider)
    maxconn: 3000                 # optional, max allowed front-end connection
    balance: roundrobin           # optional, haproxy load balance algorithm (roundrobin by default, other: leastconn)
    default_server_options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

Required

  • Name (service.name).

    The full name of the service is prefixed by the database cluster name and suffixed by service.name, connected by -.

  • Port (service.port).

    In Pigsty, services are exposed as NodePort by default, so the port is mandatory. However, if you use an LB service access scheme, you can also differentiate the services in other ways.

  • selector (service.selector).

    The selector specifies the instance members of the service, in the form of JMESPath, filtering variables from all cluster instances. The default [] selector picks all cluster members.

Optional

  • backup selector (service.selector).

    The backup selector selects or marks the list of instances for service backup, i.e., the backup instance takes over the service only when all other members of the cluster fail.

  • source_ip (service.src_ip).

    Indicates the IP used externally by the service. The default is *, which is all IP on the localhost. Using vip will use the vip_address variable to take the value, or you can also fill in the specific IP supported by the NIC.

  • Host port (service.dst_port).

    Indicates which port the service’s traffic will be directed to on the target instance. postgres will point to the port the database is listening on, pgbouncer will point to the port the connection pool is listening on, or you can fill in a fixed port.

  • health check method (service.check_method):

    How does the service check the health status of the instance? Currently, only HTTP is supported.

  • Health check port (service.check_port):

    Which port does the service check the instance on to get the health status of the instance? patroni will get it from Patroni (default 8008), pg_exporter will get it from PG Exporter (default 9630), or the user can fill in a custom port.

  • Health check path (service.check_url):

    The URL PATH is used by the service to perform HTTP checks. / is used by default for health checks, and PG Exporter and Patroni provide a variety of health check methods that can be used to differentiate between primary and replica traffic. For example, /primary will only return success for the primary, and /replica will only return success for the replica. /read-only, on the other hand, will return success for any instance that supports read-only (including the primary).

  • health check code (service.check_code):

    The code expected for HTTP health checks, default, is 200.

  • Haproxy-specific configuration (service.haproxy) :

    Proprietary config entries about the service provisioning software (HAProxy).

Service Implementation

Pigsty currently uses HAProxy-based service implementation by default and provides a sample implementation based on Layer 4 LB (L4VIP). For details, please refer to the section access.

Access

Access is designed to address high concurrency, HA, and high performance in production envs. Individual users can choose to ignore the access method and access the database directly via IP.

Access default database via postgres://dbuser_dba:[email protected]:5432/meta (replace IP & password)

In Pigsty’s default config, a fully functional LB(HAProxy) is deployed on each database instance/node, so that any instance can serve as an access point for the entire cluster. The delivery boundary of a Pigsty cluster stops at the access layer LB(HAProxy); it is up to you to decide Access Policy: how to distribute business traffic to one, multiple, or all load balancing instances.

Pigsty provides a rich set of access methods. The Pigsty sandbox uses an L2 VIP bound to the primary and a domain name attached to that VIP as a sample. The application accesses the load-balancing instance on the primary through the L2 VIP via the domain name. When this node becomes unavailable, the VIP is transferred with the primary, and the traffic is carried by the LB on the new primary, as shown in the following figure.

Another classic policy is to use DNS polling to resolve DNS domain names to all instances, and several common access patterns will be given in this article.

User Interface

The interface that Pigsty ultimately delivers to the user is a database connection string.

The formal difference between the different access methods is the difference between host and port in the connection string.

Port

Pigsty uses different ports to distinguish between database services, which provide Postgres equivalent services, as follows

port service type description
5432 postgres database Direct access to the current node database
6432 pgbouncer connection pool Accessing the current node database through a connection pool
5433 primary service Load-balancing and accessing the primary through a connection pool
5434 replica service Load-balancing and accessing the primary through a connection pool
5436 default service Direct access to the primary via load balancing
5438 offline service Direct access to the offline via load balancing

Host

type sample description
Cluster domain name pg-test Direct access to the current node database
Cluster VIP 10.10.10.3 Access the current node database through a connection pool
Instance-specific domain name pg-test-1 Load-balancing and accessing the primary through a connection pool
Instance-specific IP 10.10.10.11 Load-balancing and accessing the primary through a connection pool
All IP 10.10,10.11,10.12 Use Multihost feature

Depending on the contents of the host section and the available port, multiple connection strings can be combined.

Available Combinations

The following connection strings are available for the test database on the cluster pg-test in a singleton sandbox.

Available Combinations 
# Access via cluster domain
postgres://test@pg-test:5432/test # DNS -> L2 VIP -> primary direct connection
postgres://test@pg-test:6432/test # DNS -> L2 VIP -> primary connection pool -> primary
postgres://test@pg-test:5433/test # DNS -> L2 VIP -> HAProxy -> Primary Connection Pool -> Primary
postgres://test@pg-test:5434/test # DNS -> L2 VIP -> HAProxy -> Replica Connection Pool -> Replica
postgres://dbuser_dba@pg-test:5436/test # DNS -> L2 VIP -> HAProxy -> Primary direct connection (for Admin)
postgres://dbuser_stats@pg-test:5438/test # DNS -> L2 VIP -> HAProxy -> offline direct connection (for ETL/personal queries)

# Direct access via cluster VIP
postgres://[email protected]:5432/test # L2 VIP -> Primary direct access
postgres://[email protected]:6432/test # L2 VIP -> Primary Connection Pool -> Primary
postgres://[email protected]:5433/test # L2 VIP -> HAProxy -> Primary Connection Pool -> Primary
postgres://[email protected]:5434/test # L2 VIP -> HAProxy -> Repilca Connection Pool -> Replica
postgres://[email protected]:5436/test # L2 VIP -> HAProxy -> Primary direct connection (for Admin)
postgres://[email protected]:5438/test # L2 VIP -> HAProxy -> offline direct connect (for ETL/personal queries)

# Specify any cluster instance name directly
postgres://test@pg-test-1:5432/test # DNS -> Database Instance Direct Connect (singleton access)
postgres://test@pg-test-1:6432/test # DNS -> connection pool -> database
postgres://test@pg-test-1:5433/test # DNS -> HAProxy -> connection pool -> database read/write
postgres://test@pg-test-1:5434/test # DNS -> HAProxy -> connection pool -> database read-only
postgres://dbuser_dba@pg-test-1:5436/test # DNS -> HAProxy -> database direct connect
postgres://dbuser_stats@pg-test-1:5438/test # DNS -> HAProxy -> database offline read/write

# Directly specify any cluster instance IP access
postgres://[email protected]:5432/test # Database instance direct connection (directly specify instance, no automatic traffic distribution)
postgres://[email protected]:6432/test # Connection Pool -> Database
postgres://[email protected]:5433/test # HAProxy -> connection pool -> database read/write
postgres://[email protected]:5434/test # HAProxy -> connection pool -> database read-only
postgres://[email protected]:5436/test # HAProxy -> Database Direct Connections
postgres://[email protected]:5438/test # HAProxy -> database offline read-write

# Directly specify any cluster instance IP access
postgres://[email protected]:5432/test # Database instance direct connection (directly specify instance, no automatic traffic distribution)
postgres://[email protected]:6432/test # Connection pool -> database
postgres://[email protected]:5433/test # HAProxy -> connection pool -> database read/write
postgres://[email protected]:5434/test # HAProxy -> connection pool -> database read-only
postgres://[email protected]:5436/test # HAProxy -> Database Direct Connections
postgres://[email protected]:5438/test # HAProxy -> database offline read-write

# Smart client automatic read/write separation (connection pooling)
postgres://[email protected]:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=primary
postgres://[email protected]:6432,10.10.10.12:6432,10.10.10.13:6432/test?target_session_attrs=prefer-standby

# Intelligent client automatic read/write separation (database)
postgres://[email protected]:5432,10.10.10.12:5432,10.10.10.13:5432/test?target_session_attrs=primary
postgres://[email protected]:5432,10.10.10.12:5432,10.10.10.13:5432/test?target_session_attrs=prefer-standby

At the cluster level, users can access the four default services provided by the cluster via cluster domain + service port. Users can also bypass the domain name and access the database cluster directly using the cluster’s VIP (L2 or L4).

At the instance level, users can connect directly to Postgres via the node IP/domain name + port 5432 or port 6432 to access the database via Pgbouncer. Services provided by the cluster to which the instance belongs can also be accessed via Haproxy via 5433~543x.

Access Method

Pigsty recommends using a Haproxy-based access solution (1/2) or, in production envs with infra support, an L4VIP (or equivalent load balancing service) based access solution (3).

Serial Number Solution Description
1 L2VIP + Haproxy Standard access policy, using L2 VIP to ensure HA of Haproxy
2 DNS + Haproxy Standard HA access policy, no single node of system.
3 L4VIP + Haproxy A variant of Method 2, using L4 VIP to ensure Haprxoy is HA.
4 L4 VIP Large-scale high-performance production envs DPVS L4 VIP access is recommended
5 Consul DNS Use Consul DNS for service discovery, bypassing VIPs and Haproxy
6 Static DNS Traditional static DNS Access
7 IP Using Smart Client Access

L2 VIP + Haproxy

Solution Description

The standard access method for Pigsty sandboxes uses a single domain name bound to a single L2 VIP, which points to the HAProxy.

The Haproxy uses Node Port to expose service in a unified way. Each Haproxy is an idempotent instance, providing complete load balancing and service distribution. Haproxy is deployed on each database node so that each member of the entire cluster is idempotent.

The availability of Haproxy is achieved through idempotent replicas. Each Haproxy can be used as an access portal, and users can use one, two, or more Haproxy instances, each providing exactly functionality.

Each cluster is assigned one L2 VIP, which is fixedly bound to the primary. When a switchover of the primary occurs, that L2 VIP is moved to the new primary with it. This is achieved through vip-manager: vip-manager will query Consul to get the current primary information, and then listen to the VIP address on the primary.

The L2 VIP of the cluster has a domain name corresponding to it. The domain name is fixed to resolve to that L2 VIP and remains unchanged during the lifecycle.

Solution Superiority

  • No single point, HA

  • VIP fixed binding to the primary, can be flexible access.

Solution limitations

  • One more hop

  • A client’s IP is lost, and some HBA policies cannot take effect normally.

  • All candidate primary must be located in the same Layer 2 network.

    • As an alternative, users can also bypass this restriction by using L4 VIP, but there will be one extra hop compared to L2 VIP.
    • As an alternative, users can also choose not to use L2 VIP and use DNS to point directly to HAProxy, but may be affected by client DNS caching.

Schematic of the solution

DNS + Haproxy

Solution Description

Standard HA access method with no single point. A good balance of flexibility, applicability, and performance is achieved.

Haproxy in a cluster uses Node Port to expose service in a unified way. Each Haproxy is idempotent, providing complete load balancing and service distribution. Haproxy is deployed on each database node so that each member of the entire cluster is idempotent.

The availability of Haproxy is achieved through idempotent replicas. Each Haproxy can be used as an access portal, and users can use one, two, or multiple Haproxy instances, each providing precisely the same functionality.

The user needs to ensure that the application can access any healthy Haproxy instances. Users can resolve the DNS domain name of the database cluster to several Haproxy instances and enable DNS polling responses. And the client can choose not to cache DNS or use long connections and implement a mechanism to retry after a failed connection is established. Or refer to Method 2 and ensure HA of Haproxy with other L2/L4 VIPs on the architecture side.

Solution Superiority

  • No single point, HA

  • VIP fixed binding to the primary can be flexible access

Solution limitations

  • One more hop

  • A client’s IP is lost, and some HBA policies can not take effect properly.

  • Haproxy is HA through idempotent replica, DNS polling, and client reconnection.

    DNS should have a polling mechanism, clients should use long connections, and a failure retry mechanism should be in place. So that Haproxy failures can be transferred to other Haproxy instances in the cluster.

Schematic of the solution

L4 VIP + Haproxy

Four-layer load balancing + HAProxy access

Solution overview

Another variant of access method 1/2, ensuring HA of Haproxy via L4 VIP.

Solution advantages

  • No single point, HA.
  • Can use all Haproxy instances simultaneously to carry traffic evenly
  • All candidate primary does not need to be located in the same Layer 2 network.
  • Can operate a single VIP to complete traffic switching.

Solution limitations

  • You can use Method 4: L4 VIP direct access for two more hops, which is more wasteful.
  • The client’s IP is lost; part of the HBA policy can not correctly take effect.

L4 VIP

Four-layer load-balanced access

Program Description

Large-scale high-performance production env recommended using L4 VIP access (FullNAT, DPVS).

Solution Superiority

  • Good performance and high throughput
  • The correct client IP can be obtained through the toa module, and HBA can be fully effective.

Solution limitation

  • Still one more article.
  • We need to rely on external infra, which is complicated to deploy.
  • Still lose client IP when the toa kernel module is not enabled.
  • No Haproxy to mask primary-replica differences, and each node is no longer “idempotent”.

Consul DNS

Consul DNS access

Solution Description

The L2 VIP method is unavailable when all candidate primary must be located on the same Layer 2 network. In this case, DNS resolution can be used instead of L2 VIP

Solution Superiority

  • One less hop

Solution Limitations

  • Reliance on Consul DNS
  • User needs to configure DNS caching policy properly

Static DNS

Static DNS Access

Solution Introduction

Traditional static DNS access method

Advantages of the solution

  • One less hop
  • Simple implementation

Solution Limitations

  • No flexibility
  • Prone to traffic loss during primary-replica switching

IP

IP Direct Access

Solution Introduction

Direct database IP access using innovative clients

Solution advantages

  • Direct connection to database/connection pool, one less
  • No reliance on additional components for primary-replica differentiation, reducing system complexity.

Solution limitations

  • Too inflexible, cumbersome to expand and reduce cluster capacity.

12 - PGSQL Authentication and Privilege

Historical documentation restored from the Pigsty v1.5.1 tag.

PostgreSQL provides a standard access control mechanism: Authentication and Privileges, both of which are based on the Role system.

Role

Pigsty’s default role system contains four default roles and four default users：

name attr roles desc
dbrole_readonly Cannot login role for global readonly access
dbrole_readwrite Cannot login dbrole_readonly role for global read-write access
dbrole_offline Cannot login role for restricted read-only access (offline instance)
dbrole_admin Cannot login
Bypass RLS		 pg_monitor
pg_signal_backend
dbrole_readwrite		 role for object creation
postgres Superuser
Create role
Create DB
Replication
Bypass RLS		 system superuser
replicator Replication
Bypass RLS		 pg_monitor
dbrole_readonly		 system replicator
dbuser_monitor 16 connections pg_monitor
dbrole_readonly		 system monitor user
dbuser_dba Bypass RLS
Superuser		 dbrole_admin system admin user

Default Roles

Pigsty has four default roles：

  • Read-only role (dbrole_readonly): Has read-only access to all data tables.
  • Read-write role (dbrole_readwrite): Has to write access to all data tables, inherits dbrole_readonly.
  • Admin role (dbrole_admin): Can execute DDL changes, inherits dbrole_readwrite.
  • Offline role (dbrole_offline): A special read-only role for executing slow queries/ETL/interactive queries, only allowed access to specific instances.

The definition is shown below.

- { name: dbrole_readonly  , login: false , comment: role for global read-only access  }                            # production read-only role
- { name: dbrole_offline ,   login: false , comment: role for restricted read-only access (offline instance) }      # restricted-read-only role
- { name: dbrole_readwrite , login: false , roles: [dbrole_readonly], comment: role for global read-write access }  # production read-write role
- { name: dbrole_admin , login: false , roles: [pg_monitor, dbrole_readwrite] , comment: role for object creation } # production DDL change role
Warning

Common users should not change the name of the default role.

Default Users

Pigsty has four default users.

  • superuser (postgres), the owner and creator of the database, the same as the OS user.
  • Replication user (replicator), the system user used for primary-replica.
  • Monitor user (dbuser_monitor), a user used to monitor database and connection pool metrics.
  • Admin user (dbuser_dba), the admin user who performs daily operations and database changes.

The definitions are shown below:

- { name: postgres , superuser: true , comment: system superuser }                             # system dbsu, name is designated by `pg_dbsu`
- { name: dbuser_dba , superuser: true , roles: [dbrole_admin] , comment: system admin user }  # admin dbsu, name is designated by `pg_admin_username`
- { name: replicator , replication: true , bypassrls: true , roles: [pg_monitor, dbrole_readonly] , comment: system replicator }                   # replicator
- { name: dbuser_monitor , roles: [pg_monitor, dbrole_readonly] , comment: system monitor user , parameters: {log_min_duration_statement: 1000 } } # monitor user

In Pigsty, four important default usernames and passwords are controlled and managed by separate parameters.

pg_dbsu: postgres                             # os user for the database

# - system roles - #
pg_replication_username: replicator           # system replication user
pg_replication_password: DBUser.Replicator    # system replication password
pg_monitor_username: dbuser_monitor           # system monitor user
pg_monitor_password: DBUser.Monitor           # system monitor password
pg_admin_username: dbuser_dba                 # system admin user
pg_admin_password: DBUser.DBA                 # system admin password

It is not recommended to set a password or allow remote access for the default superuser postgres, so there is no dedicated dbsu_password option. If there is such a need, you can set a password for the dbsu in pg_default_roles.

Warning

Be sure to change the passwords of all default users.

In addition, users can define cluster-specific business users in pg_users in the same way as pg_default_roles.

Warning

It is recommended to remove the dborle_readony role from dbuser_monitor if there is a higher data security requirement. Some of the monitoring system features will not be available.

Authentication

Pigsty uses md5 password authentication by default and provides access control based on the PostgreSQL HBA mechanism.

HBA(Host Based Authentication)can be treated as an IP blocklist and allowlist.

Config: HBA

In Pigsty, the HBA of all instances is generated from the config file, and HBA rules vary depending on the instance’s role (pg_role). The following variables control pigsty’s HBAs.

Each variable is an array consisting of the following rules.

- title: allow intranet admin password access
  role: common
  rules:
    - host    all     +dbrole_admin               10.0.0.0/8          md5
    - host    all     +dbrole_admin               172.16.0.0/12       md5
    - host    all     +dbrole_admin               192.168.0.0/16      md5

Role-Based HBA

The HBA rule set with role = common is installed to all instances,(role: primary) are only installed to instances with pg_role = primary.

As a special case, the HBA rule for the role: offline will be installed to instances with pg_role == 'offline' as well as to instances with pg_offline_query == true.

The rendering priority rules for HBA are:

  • hard_coded_rules Global hard-coded rules
  • pg_hba_rules_extra.common Cluster common rules
  • pg_hba_rules_extra.pg_role Cluster role rules
  • pg_hba_rules.pg_role Global role rules
  • pg_hba_rules.offline Cluster offline rules
  • pg_hba_rules_extra.offline Global offline rules
  • pg_hba_rules.common Global common rules

Default HBA Rules

Under the default config, the primary and replica will use the following HBA rules:

  • Superuser access with local OS auth.
  • Other users can access it with a password from local.
  • Replica users can access via password from the LAN segment.
  • Monitor users can access it locally.
  • Everyone can access it with a password on the meta node.
  • Admin users can access via password from the LAN.
  • Everyone can access the intranet with a password.
  • Read and write users (production business users) can be accessed locally (Connection Pool).
  • On the replica: read-only users (individuals) can access from the local (Connection Pool).
  • On instances with pg_role == 'offline' or with pg_offline_query == true, HBA rules that allow access to dbrole_offline grouped users are added.
Default HBA rule information 
#==============================================================#
# Default HBA
#==============================================================#
# allow local su with ident"
local   all             postgres                               ident
local   replication     postgres                               ident

# allow local user password access
local   all             all                                    md5

# allow local/intranet replication with password
local   replication     replicator                              md5
host    replication     replicator         127.0.0.1/32         md5
host    all             replicator         10.0.0.0/8           md5
host    all             replicator         172.16.0.0/12        md5
host    all             replicator         192.168.0.0/16       md5
host    replication     replicator         10.0.0.0/8           md5
host    replication     replicator         172.16.0.0/12        md5
host    replication     replicator         192.168.0.0/16       md5

# allow local role monitor with password
local   all             dbuser_monitor                          md5
host    all             dbuser_monitor      127.0.0.1/32        md5

#==============================================================#
# Extra HBA
#==============================================================#
# add extra hba rules here

#==============================================================#
# primary HBA
#==============================================================#

#==============================================================#
# special HBA for instance marked with 'pg_offline_query = true'
#==============================================================#

#==============================================================#
# Common HBA
#==============================================================#
#  allow meta node password access
host    all     all                         10.10.10.10/32      md5

#  allow intranet admin password access
host    all     +dbrole_admin               10.0.0.0/8          md5
host    all     +dbrole_admin               172.16.0.0/12       md5
host    all     +dbrole_admin               192.168.0.0/16      md5

#  allow intranet password access
host    all             all                 10.0.0.0/8          md5
host    all             all                 172.16.0.0/12       md5
host    all             all                 192.168.0.0/16      md5

#  allow local read/write (local production user via pgbouncer)
local   all     +dbrole_readonly                                md5
host    all     +dbrole_readonly           127.0.0.1/32         md5

#==============================================================#
# Ad Hoc HBA
#===========================================================

Change HBA Rules

Users can modify and apply the new HBA rules through a playbook after the cluster/instance is created and running.

./pgsql.yml -t pg_hba    # Specify the target cluster with -l
bin/reloadhba <cluster>  # Reload the HBA rules

When the database cluster directory is destroyed and rebuilt, the new copy will have the same HBA rules as the cluster primary. You can use the above command to perform HBA repair for a specific instance.

Pgbouncer HBA

In Pigsty, Pgbouncer also uses HBA for access control. The usage is the same as Postgres HBA:

The default Pgbouncer HBA rules allow password access from local and intranet.

pgbouncer_hba_rules:                          # pgbouncer host-based authentication rules
  - title: local password access
    role: common
    rules:
      - local  all          all                                     md5
      - host   all          all                     127.0.0.1/32    md5

  - title: intranet password access
    role: common
    rules:
      - host   all          all                     10.0.0.0/8      md5
      - host   all          all                     172.16.0.0/12   md5
      - host   all          all                     192.168.0.0/16  md5

Privilege

Pigsty’s default privilege model is related to the default role. When using the Pigsty access control, all newly created business users should belong to one of the four default roles, which have the privileges shown below:

  • All users have access to all schemas.
  • Read-only users can read all tables.
  • Read-write users can perform DML operations (INSERT, UPDATE, DELETE).
  • Admin users can perform DDL change operations (CREATE, USAGE, TRUNCATE, REFERENCES, TRIGGER).
  • Offline and read-only users are only allowed to access instances of pg_role == 'offline' or pg_offline_query = true.
GRANT USAGE                         ON SCHEMAS   TO dbrole_readonly;
GRANT SELECT                        ON TABLES    TO dbrole_readonly;
GRANT SELECT                        ON SEQUENCES TO dbrole_readonly;
GRANT EXECUTE                       ON FUNCTIONS TO dbrole_readonly;
GRANT USAGE                         ON SCHEMAS   TO dbrole_offline;
GRANT SELECT                        ON TABLES    TO dbrole_offline;
GRANT SELECT                        ON SEQUENCES TO dbrole_offline;
GRANT EXECUTE                       ON FUNCTIONS TO dbrole_readonly;
GRANT INSERT, UPDATE, DELETE        ON TABLES    TO dbrole_readwrite;
GRANT USAGE,  UPDATE                ON SEQUENCES TO dbrole_readwrite;
GRANT TRUNCATE, REFERENCES, TRIGGER ON TABLES    TO dbrole_admin;
GRANT CREATE                        ON SCHEMAS   TO dbrole_admin;
GRANT USAGE                         ON TYPES     TO dbrole_admin;
Owner Schema Type Access privileges
username schema postgres=UC/postgres
dbrole_readonly=U/postgres
dbrole_offline=U/postgres
dbrole_admin=C/postgres
username sequence postgres=rwU/postgres
dbrole_readonly=r/postgres
dbrole_readwrite=wU/postgres
dbrole_offline=r/postgres
username table postgres=arwdDxt/postgres
dbrole_readonly=r/postgres
dbrole_readwrite=awd/postgres
dbrole_offline=r/postgres
dbrole_admin=Dxt/postgres
username function =X/postgres
postgres=X/postgres
dbrole_readonly=X/postgres
dbrole_offline=X/postgres

Privilege Maintenance

PostgreSQL’s ALTER DEFAULT PRIVILEGES ensures default access to database objects.

All objects created by {{ dbsu }}, {{ pg_admin_username }}, {{ dbrole_admin }} will have the default privileges.

PostgreSQL’s ALTER DEFAULT PRIVILEGE only takes effect for “objects created by specific users” objects created by superuser postgres, and dbuser_dba have default privileges. Suppose you want to give business users privileges to execute DDL besides giving the dbrole_admin role to business users. You should also remember that you should first run the following command when executing DDL changes.

SET ROLE dbrole_admin; -- dbrole_admin creates objects with the correct default permissions

Database Privileges

The database has three privileges: CONNECT, CREATE, TEMP, and a special genus OWNERSHIP. The parameter pg_database controls the definition of the database. A complete database definition is shown below:

pg_databases:                       # define business databases on this cluster, array of database definition
  # define the default `meta` database
  - name: meta                      # required, `name` is the only mandatory field of a database definition
    baseline: cmdb.sql              # optional, database sql baseline path, (relative path among ansible search path, e.g files/)
    owner: postgres                 # optional, database owner, postgres by default
    template: template1             # optional, which template to use, template1 by default
    encoding: UTF8                  # optional, database encoding, UTF8 by default. (MUST same as template database)
    locale: C                       # optional, database locale, C by default.  (MUST same as template database)
    lc_collate: C                   # optional, database collate, C by default. (MUST same as template database)
    lc_ctype: C                     # optional, database ctype, C by default.   (MUST same as template database)
    tablespace: pg_default          # optional, default tablespace, 'pg_default' by default.
    allowconn: true                 # optional, allow connection, true by default. false will disable connect at all
    revokeconn: false               # optional, revoke public connection privilege. false by default. (leave connect with grant option to owner)
    pgbouncer: true                 # optional, add this database to pgbouncer database list? true by default
    comment: pigsty meta database   # optional, comment string for this database
    connlimit: -1                   # optional, database connection limit, default -1 disable limit
    schemas: [pigsty]               # optional, additional schemas to be created, array of schema names
    extensions:                     # optional, additional extensions to be installed: array of schema definition `{name,schema}`
      - {name: adminpack, schema: pg_catalog}    # install adminpack to pg_catalog and install postgis to public
      - {name: postgis, schema: public}          # if schema is omitted, extension will be installed according to search_path.

If the database is not configured with an owner, dbsu will be the default OWNER of the database. Otherwise, it will be the specified user.

All users have the CONNECT privilege to the newly created database; set revokeconn == true if you wish to reclaim this privilege. Only the default user (dbsu|admin|monitor|replicator) with the database’s owner is explicitly given the CONNECT privilege. Also, admin|owner will have GRANT OPTION for the CONNECT privilege and can transfer the CONNECT privilege to others.

If you implement access isolation between different databases, you can create a business user as the owner for each database and set the revokeconn option for all of them.

A sample database for privilege isolation 
#--------------------------------------------------------------#
# pg-infra (example database for cluster loading)
#--------------------------------------------------------------#
pg-infra:
  hosts:
    10.10.10.40: { pg_seq: 1, pg_role: primary }
    10.10.10.41: { pg_seq: 2, pg_role: replica , pg_offline_query: true }
  vars:
    pg_cluster: pg-infrastructure
    pg_version: 14
    vip_address: 10.10.10.4
    pgbouncer_poolmode: session
    pg_hba_rules_extra:
      - title: allow confluence jira gitlab eazybi direct access
        role: common
        rules:
          - host    confluence dbuser_confluence   10.0.0.0/8        md5
          - host    jira       dbuser_jira         10.0.0.0/8        md5
          - host    gitlab     dbuser_gitlab       10.0.0.0/8        md5

    pg_users:
      # infra prod user
      - { name: dbuser_hybridcloud, password: ssag-2xd, pgbouncer: true, roles: [ dbrole_readwrite ] }
      - { name: dbuser_confluence, password: mc2iohos , pgbouncer: true, roles: [ dbrole_admin ] }
      - { name: dbuser_gitlab, password: sdf23g22sfdd , pgbouncer: true, roles: [ dbrole_readwrite ] }
      - { name: dbuser_jira, password: sdpijfsfdsfdfs , pgbouncer: true, roles: [ dbrole_admin ] }
    pg_databases:
      # infra database
      - { name: hybridcloud , revokeconn: true, owner: dbuser_hybridcloud , parameters: { search_path: yay,public } , connlimit: 100 }
      - { name: confluence , revokeconn: true, owner: dbuser_confluence , connlimit: 100 }
      - { name: gitlab , revokeconn: true, owner: dbuser_gitlab, connlimit: 100 }
      - { name: jira , revokeconn: true, owner: dbuser_jira , connlimit: 100 }

Create Privilege

Pigsty revokes the PUBLIC user’s privilege to CREATE a new schema under the database for security reasons. It also revokes the PUBLIC user’s privilege to create new relationships in the PUBLIC schema. The database superuser and admin user are not subject to this restriction.

Privileges to create objects in the database are independent of whether the user is the database owner or not. It only depends on whether the user was given admin privileges when it was created.

pg_users:
  - {name: test1, password: xxx , groups: [dbrole_readwrite]}  # Schema with objects cannot be created
  - {name: test2, password: xxx , groups: [dbrole_admin]}      # Schema and objects can be created

13 - PGSQL Authentication and Privilege

Historical documentation restored from the Pigsty v1.5.1 tag.

PostgreSQL provides a standard access control mechanism: Authentication and Privileges, both of which are based on the Role system.

Role

Pigsty’s default role system contains four default roles and four default users：

name attr roles desc
dbrole_readonly Cannot login role for global readonly access
dbrole_readwrite Cannot login dbrole_readonly role for global read-write access
dbrole_offline Cannot login role for restricted read-only access (offline instance)
dbrole_admin Cannot login
Bypass RLS		 pg_monitor
pg_signal_backend
dbrole_readwrite		 role for object creation
postgres Superuser
Create role
Create DB
Replication
Bypass RLS		 system superuser
replicator Replication
Bypass RLS		 pg_monitor
dbrole_readonly		 system replicator
dbuser_monitor 16 connections pg_monitor
dbrole_readonly		 system monitor user
dbuser_dba Bypass RLS
Superuser		 dbrole_admin system admin user

Default Roles

Pigsty has four default roles：

  • Read-only role (dbrole_readonly): Has read-only access to all data tables.
  • Read-write role (dbrole_readwrite): Has to write access to all data tables, inherits dbrole_readonly.
  • Admin role (dbrole_admin): Can execute DDL changes, inherits dbrole_readwrite.
  • Offline role (dbrole_offline): A special read-only role for executing slow queries/ETL/interactive queries, only allowed access to specific instances.

The definition is shown below.

- { name: dbrole_readonly  , login: false , comment: role for global read-only access  }                            # production read-only role
- { name: dbrole_offline ,   login: false , comment: role for restricted read-only access (offline instance) }      # restricted-read-only role
- { name: dbrole_readwrite , login: false , roles: [dbrole_readonly], comment: role for global read-write access }  # production read-write role
- { name: dbrole_admin , login: false , roles: [pg_monitor, dbrole_readwrite] , comment: role for object creation } # production DDL change role
Warning

Common users should not change the name of the default role.

Default Users

Pigsty has four default users.

  • superuser (postgres), the owner and creator of the database, the same as the OS user.
  • Replication user (replicator), the system user used for primary-replica.
  • Monitor user (dbuser_monitor), a user used to monitor database and connection pool metrics.
  • Admin user (dbuser_dba), the admin user who performs daily operations and database changes.

The definitions are shown below:

- { name: postgres , superuser: true , comment: system superuser }                             # system dbsu, name is designated by `pg_dbsu`
- { name: dbuser_dba , superuser: true , roles: [dbrole_admin] , comment: system admin user }  # admin dbsu, name is designated by `pg_admin_username`
- { name: replicator , replication: true , bypassrls: true , roles: [pg_monitor, dbrole_readonly] , comment: system replicator }                   # replicator
- { name: dbuser_monitor , roles: [pg_monitor, dbrole_readonly] , comment: system monitor user , parameters: {log_min_duration_statement: 1000 } } # monitor user

In Pigsty, four important default usernames and passwords are controlled and managed by separate parameters.

pg_dbsu: postgres                             # os user for the database

# - system roles - #
pg_replication_username: replicator           # system replication user
pg_replication_password: DBUser.Replicator    # system replication password
pg_monitor_username: dbuser_monitor           # system monitor user
pg_monitor_password: DBUser.Monitor           # system monitor password
pg_admin_username: dbuser_dba                 # system admin user
pg_admin_password: DBUser.DBA                 # system admin password

It is not recommended to set a password or allow remote access for the default superuser postgres, so there is no dedicated dbsu_password option. If there is such a need, you can set a password for the dbsu in pg_default_roles.

Warning

Be sure to change the passwords of all default users.

In addition, users can define cluster-specific business users in pg_users in the same way as pg_default_roles.

Warning

It is recommended to remove the dborle_readony role from dbuser_monitor if there is a higher data security requirement. Some of the monitoring system features will not be available.

Authentication

Pigsty uses md5 password authentication by default and provides access control based on the PostgreSQL HBA mechanism.

HBA(Host Based Authentication)can be treated as an IP blocklist and allowlist.

Config: HBA

In Pigsty, the HBA of all instances is generated from the config file, and HBA rules vary depending on the instance’s role (pg_role). The following variables control pigsty’s HBAs.

Each variable is an array consisting of the following rules.

- title: allow intranet admin password access
  role: common
  rules:
    - host    all     +dbrole_admin               10.0.0.0/8          md5
    - host    all     +dbrole_admin               172.16.0.0/12       md5
    - host    all     +dbrole_admin               192.168.0.0/16      md5

Role-Based HBA

The HBA rule set with role = common is installed to all instances,(role: primary) are only installed to instances with pg_role = primary.

As a special case, the HBA rule for the role: offline will be installed to instances with pg_role == 'offline' as well as to instances with pg_offline_query == true.

The rendering priority rules for HBA are:

  • hard_coded_rules Global hard-coded rules
  • pg_hba_rules_extra.common Cluster common rules
  • pg_hba_rules_extra.pg_role Cluster role rules
  • pg_hba_rules.pg_role Global role rules
  • pg_hba_rules.offline Cluster offline rules
  • pg_hba_rules_extra.offline Global offline rules
  • pg_hba_rules.common Global common rules

Default HBA Rules

Under the default config, the primary and replica will use the following HBA rules:

  • Superuser access with local OS auth.
  • Other users can access it with a password from local.
  • Replica users can access via password from the LAN segment.
  • Monitor users can access it locally.
  • Everyone can access it with a password on the meta node.
  • Admin users can access via password from the LAN.
  • Everyone can access the intranet with a password.
  • Read and write users (production business users) can be accessed locally (Connection Pool).
  • On the replica: read-only users (individuals) can access from the local (Connection Pool).
  • On instances with pg_role == 'offline' or with pg_offline_query == true, HBA rules that allow access to dbrole_offline grouped users are added.
Default HBA rule information 
#==============================================================#
# Default HBA
#==============================================================#
# allow local su with ident"
local   all             postgres                               ident
local   replication     postgres                               ident

# allow local user password access
local   all             all                                    md5

# allow local/intranet replication with password
local   replication     replicator                              md5
host    replication     replicator         127.0.0.1/32         md5
host    all             replicator         10.0.0.0/8           md5
host    all             replicator         172.16.0.0/12        md5
host    all             replicator         192.168.0.0/16       md5
host    replication     replicator         10.0.0.0/8           md5
host    replication     replicator         172.16.0.0/12        md5
host    replication     replicator         192.168.0.0/16       md5

# allow local role monitor with password
local   all             dbuser_monitor                          md5
host    all             dbuser_monitor      127.0.0.1/32        md5

#==============================================================#
# Extra HBA
#==============================================================#
# add extra hba rules here

#==============================================================#
# primary HBA
#==============================================================#

#==============================================================#
# special HBA for instance marked with 'pg_offline_query = true'
#==============================================================#

#==============================================================#
# Common HBA
#==============================================================#
#  allow meta node password access
host    all     all                         10.10.10.10/32      md5

#  allow intranet admin password access
host    all     +dbrole_admin               10.0.0.0/8          md5
host    all     +dbrole_admin               172.16.0.0/12       md5
host    all     +dbrole_admin               192.168.0.0/16      md5

#  allow intranet password access
host    all             all                 10.0.0.0/8          md5
host    all             all                 172.16.0.0/12       md5
host    all             all                 192.168.0.0/16      md5

#  allow local read/write (local production user via pgbouncer)
local   all     +dbrole_readonly                                md5
host    all     +dbrole_readonly           127.0.0.1/32         md5

#==============================================================#
# Ad Hoc HBA
#===========================================================

Change HBA Rules

Users can modify and apply the new HBA rules through a playbook after the cluster/instance is created and running.

./pgsql.yml -t pg_hba    # Specify the target cluster with -l
bin/reloadhba <cluster>  # Reload the HBA rules

When the database cluster directory is destroyed and rebuilt, the new copy will have the same HBA rules as the cluster primary. You can use the above command to perform HBA repair for a specific instance.

Pgbouncer HBA

In Pigsty, Pgbouncer also uses HBA for access control. The usage is the same as Postgres HBA:

The default Pgbouncer HBA rules allow password access from local and intranet.

pgbouncer_hba_rules:                          # pgbouncer host-based authentication rules
  - title: local password access
    role: common
    rules:
      - local  all          all                                     md5
      - host   all          all                     127.0.0.1/32    md5

  - title: intranet password access
    role: common
    rules:
      - host   all          all                     10.0.0.0/8      md5
      - host   all          all                     172.16.0.0/12   md5
      - host   all          all                     192.168.0.0/16  md5

Privilege

Pigsty’s default privilege model is related to the default role. When using the Pigsty access control, all newly created business users should belong to one of the four default roles, which have the privileges shown below:

  • All users have access to all schemas.
  • Read-only users can read all tables.
  • Read-write users can perform DML operations (INSERT, UPDATE, DELETE).
  • Admin users can perform DDL change operations (CREATE, USAGE, TRUNCATE, REFERENCES, TRIGGER).
  • Offline and read-only users are only allowed to access instances of pg_role == 'offline' or pg_offline_query = true.
GRANT USAGE                         ON SCHEMAS   TO dbrole_readonly;
GRANT SELECT                        ON TABLES    TO dbrole_readonly;
GRANT SELECT                        ON SEQUENCES TO dbrole_readonly;
GRANT EXECUTE                       ON FUNCTIONS TO dbrole_readonly;
GRANT USAGE                         ON SCHEMAS   TO dbrole_offline;
GRANT SELECT                        ON TABLES    TO dbrole_offline;
GRANT SELECT                        ON SEQUENCES TO dbrole_offline;
GRANT EXECUTE                       ON FUNCTIONS TO dbrole_readonly;
GRANT INSERT, UPDATE, DELETE        ON TABLES    TO dbrole_readwrite;
GRANT USAGE,  UPDATE                ON SEQUENCES TO dbrole_readwrite;
GRANT TRUNCATE, REFERENCES, TRIGGER ON TABLES    TO dbrole_admin;
GRANT CREATE                        ON SCHEMAS   TO dbrole_admin;
GRANT USAGE                         ON TYPES     TO dbrole_admin;
Owner Schema Type Access privileges
username schema postgres=UC/postgres
dbrole_readonly=U/postgres
dbrole_offline=U/postgres
dbrole_admin=C/postgres
username sequence postgres=rwU/postgres
dbrole_readonly=r/postgres
dbrole_readwrite=wU/postgres
dbrole_offline=r/postgres
username table postgres=arwdDxt/postgres
dbrole_readonly=r/postgres
dbrole_readwrite=awd/postgres
dbrole_offline=r/postgres
dbrole_admin=Dxt/postgres
username function =X/postgres
postgres=X/postgres
dbrole_readonly=X/postgres
dbrole_offline=X/postgres

Privilege Maintenance

PostgreSQL’s ALTER DEFAULT PRIVILEGES ensures default access to database objects.

All objects created by {{ dbsu }}, {{ pg_admin_username }}, {{ dbrole_admin }} will have the default privileges.

PostgreSQL’s ALTER DEFAULT PRIVILEGE only takes effect for “objects created by specific users” objects created by superuser postgres, and dbuser_dba have default privileges. Suppose you want to give business users privileges to execute DDL besides giving the dbrole_admin role to business users. You should also remember that you should first run the following command when executing DDL changes.

SET ROLE dbrole_admin; -- dbrole_admin creates objects with the correct default permissions

Database Privileges

The database has three privileges: CONNECT, CREATE, TEMP, and a special genus OWNERSHIP. The parameter pg_database controls the definition of the database. A complete database definition is shown below:

pg_databases:                       # define business databases on this cluster, array of database definition
  # define the default `meta` database
  - name: meta                      # required, `name` is the only mandatory field of a database definition
    baseline: cmdb.sql              # optional, database sql baseline path, (relative path among ansible search path, e.g files/)
    owner: postgres                 # optional, database owner, postgres by default
    template: template1             # optional, which template to use, template1 by default
    encoding: UTF8                  # optional, database encoding, UTF8 by default. (MUST same as template database)
    locale: C                       # optional, database locale, C by default.  (MUST same as template database)
    lc_collate: C                   # optional, database collate, C by default. (MUST same as template database)
    lc_ctype: C                     # optional, database ctype, C by default.   (MUST same as template database)
    tablespace: pg_default          # optional, default tablespace, 'pg_default' by default.
    allowconn: true                 # optional, allow connection, true by default. false will disable connect at all
    revokeconn: false               # optional, revoke public connection privilege. false by default. (leave connect with grant option to owner)
    pgbouncer: true                 # optional, add this database to pgbouncer database list? true by default
    comment: pigsty meta database   # optional, comment string for this database
    connlimit: -1                   # optional, database connection limit, default -1 disable limit
    schemas: [pigsty]               # optional, additional schemas to be created, array of schema names
    extensions:                     # optional, additional extensions to be installed: array of schema definition `{name,schema}`
      - {name: adminpack, schema: pg_catalog}    # install adminpack to pg_catalog and install postgis to public
      - {name: postgis, schema: public}          # if schema is omitted, extension will be installed according to search_path.

If the database is not configured with an owner, dbsu will be the default OWNER of the database. Otherwise, it will be the specified user.

All users have the CONNECT privilege to the newly created database; set revokeconn == true if you wish to reclaim this privilege. Only the default user (dbsu|admin|monitor|replicator) with the database’s owner is explicitly given the CONNECT privilege. Also, admin|owner will have GRANT OPTION for the CONNECT privilege and can transfer the CONNECT privilege to others.

If you implement access isolation between different databases, you can create a business user as the owner for each database and set the revokeconn option for all of them.

A sample database for privilege isolation 
#--------------------------------------------------------------#
# pg-infra (example database for cluster loading)
#--------------------------------------------------------------#
pg-infra:
  hosts:
    10.10.10.40: { pg_seq: 1, pg_role: primary }
    10.10.10.41: { pg_seq: 2, pg_role: replica , pg_offline_query: true }
  vars:
    pg_cluster: pg-infrastructure
    pg_version: 14
    vip_address: 10.10.10.4
    pgbouncer_poolmode: session
    pg_hba_rules_extra:
      - title: allow confluence jira gitlab eazybi direct access
        role: common
        rules:
          - host    confluence dbuser_confluence   10.0.0.0/8        md5
          - host    jira       dbuser_jira         10.0.0.0/8        md5
          - host    gitlab     dbuser_gitlab       10.0.0.0/8        md5

    pg_users:
      # infra prod user
      - { name: dbuser_hybridcloud, password: ssag-2xd, pgbouncer: true, roles: [ dbrole_readwrite ] }
      - { name: dbuser_confluence, password: mc2iohos , pgbouncer: true, roles: [ dbrole_admin ] }
      - { name: dbuser_gitlab, password: sdf23g22sfdd , pgbouncer: true, roles: [ dbrole_readwrite ] }
      - { name: dbuser_jira, password: sdpijfsfdsfdfs , pgbouncer: true, roles: [ dbrole_admin ] }
    pg_databases:
      # infra database
      - { name: hybridcloud , revokeconn: true, owner: dbuser_hybridcloud , parameters: { search_path: yay,public } , connlimit: 100 }
      - { name: confluence , revokeconn: true, owner: dbuser_confluence , connlimit: 100 }
      - { name: gitlab , revokeconn: true, owner: dbuser_gitlab, connlimit: 100 }
      - { name: jira , revokeconn: true, owner: dbuser_jira , connlimit: 100 }

Create Privilege

Pigsty revokes the PUBLIC user’s privilege to CREATE a new schema under the database for security reasons. It also revokes the PUBLIC user’s privilege to create new relationships in the PUBLIC schema. The database superuser and admin user are not subject to this restriction.

Privileges to create objects in the database are independent of whether the user is the database owner or not. It only depends on whether the user was given admin privileges when it was created.

pg_users:
  - {name: test1, password: xxx , groups: [dbrole_readwrite]}  # Schema with objects cannot be created
  - {name: test2, password: xxx , groups: [dbrole_admin]}      # Schema and objects can be created

14 - Pigsty Deployment

Historical documentation restored from the Pigsty v1.5.1 tag.

It takes three steps to deploy Pigsty： Prepare, Configure, Playbook.

Preparation

Before installing Pigsty, you need to prepare the required resources: physical/VM nodes, admin users, and download Pigsty software.

Configuration

After preparation, you need to indicate to Pigsty what infra and database services you need via configure.

Playbook Execution

The next step can be to land the requirements by executing the playbook.

Deployment

  • Standard Deployment: Prepare brand new nodes to complete the standard Pigsty deployment process.
  • Sandbox Deployment: Pull up a local VM sandbox environment with one click using a pre-built vagrant template.
  • Multi-Cloud Deployment: Use terraform template to pull up the required VM resources at the cloud service vendor and perform the deployment.
  • Monly Deployment: Use singleton Pigsty to monitor existing database clusters.

15 - Preparation

Historical documentation restored from the Pigsty v1.5.1 tag.

How to prepare the resources required for Pigsty deployment.

Node Provisioning

Before deploying Pigsty, the user needs to prepare machine node resources with arbitrary database nodes, including at least one meta node.

The nodes can use any nodes: bare metals, local VMs, cloud VMs, containers, etc… only if the following conditions are met:

  • Processor architecture: x86_64
  • Hardware specifications: 1C/1GB at least
  • Operating System: CentOS 7.8.2003 (or RHEL7 equivalent)
  • Admin User can ssh to the meta node and execute sudo commands.

One node is sufficient if you are using Pigsty as a battery-included PostgreSQL database instance. If you also plan to use Pigsty as a control for more nodes/databases, you can prepare more nodes for backup.

Meta Node Provisioning

Pigsty requires meta nodes as the admin controller of the entire environment and provides infra services.

The minimum number of meta-nodes is 1. Pigsty’s infra is deployed as replicas on multiple meta nodes, except for DCS (Consul/Etcd), which exists as Quorum.

Pigsty clusters require the use of DCS for HA functionality. You can use DCS clusters that are automatically deployed on meta nodes or use external DCS clusters. Using three meta nodes is recommended in large-scale production environments if you do not have a dedicated external DCS cluster.

Users should ensure that they can log in to the meta node and log in to other nodes via ssh with sudo or root access from the meta node using the admin user. Users should ensure access to port 80 of the meta node to access the Pigsty user interface.

  • Number of meta nodes: odd number, at least 1
  • Ability to log in to the meta node using the admin user
  • Ability to access port 80 of the meta node via browser
  • Admin users can log in to the database node remotely ssh from the meta node and execute sudo (including itself).

Admin Provisioning

Pigsty requires an admin user to SSH into other nodes from the meta node and execute sudo commands.

  • Can use this user on the meta node
  • Can SSH to all managed nodes (including itself) with this user
  • Can execute the sudo command after logging in to all managed nodes (including itself)
  • Admin user is not postgres or {{ dbsu }} (using DBSU as admin is a security risk)
  • ssh login password-free, sudo command password-free (or you know how to enter it manually via -k,-K)

The admin user you are using must have ssh and sudo privileges for all nodes when performing deployments and changes. Password-free is not required. You can always pass in ssh and sudo passwords via the -k|-K when executing the playbook or use another user to run the playbook via -eansible_host=<another_user>. However, Pigsty strongly recommends configuring SSH password-free login with password-free sudo for the admin user.

Pigsty recommends that the creation of admin users, privilege config, and key distribution be done in the Provisioning phase of the VM. For a production environment, the machine should be delivered with such a user configured with password-free remote SSH login and performing password-free sudo.

The Pigsty playbook nodes can be used to create admin users on nodes. In the Bootstrap phase, as long as you have an SSH login with SUDO access, you can use it to execute the Ansible playbook even without a password. Please refer to Nodes: Create an admin user for more details.

Manual config of SSH and SUDO

Manual config of SSH password-free login can be achieved by ssh-keygen and ssh-copy-id. Please refer to the related doc.

Manually configuring password-free sudo for a user can be done by adding the following entry to the /etc/sudoers.d/<username> file. Note that replacing <username> with the name of the administrator.

%<username> ALL=(ALL) NOPASSWD: ALL

Software Provisioning

To run Pigsty, you need to have the following software.

To run the Pigsty sandbox on your own laptop, you will also need to download and install it on the host.

  • Vagrant: VM hosting orchestration software (cross-platform, free)
  • Virtualbox: VM software (cross-platform, open-source, and free)

If you wish to run Pigsty sandbox on a cloud vendor server, you must. The. Use download and install Terraform locally.

Pigsty Source Code

Users should get the Pigsty project source on the meta node, usually unpacked to the admin user HOME dir.

# It is recommended to use this command to download the pigsty.tgz source, the script will distinguish between inside and outside the wall, use CDN to accelerate the download in mainland
curl -SL https://github.com/Vonng/pigsty/releases/download/v1.5.1/pigsty.tgz | gzip -d | tar -xC ~  # get latest pigsty source

You can also download the source tarball in other ways.

# https://github.com/Vonng/pigsty/releases/download/v1.5.1/pigsty.tgz   # Github Release
# http://download.pigsty.cc/v1.5.1/pigsty.tgz                           # China CDN
# https://pan.baidu.com/s/1DZIa9X2jAxx69Zj-aRHoaw?pwd=8su9              # Baidu Cloud Download
# git clone --branch v1.5.1 --depth 1 https://github.com/pgsty/pigsty.git                             # Clone the frozen v1.5.1 tag

Also, the download script in the root of the pigsty project can be used to download the source.

./download pigsty.tgz    # Download the current version of pigsty.tgz from Github/CDN to /tmp/pigsty.tgz
./download pigsty        # Download the current version of pigsty.tgz from Github/CDN and extract it to ~/pigsty (skip it if it already exists)

Pigsty Offline Package

The offline package packs all software packages, is about 1GB, and is optional. If /tmp/pkg.tgz already exists during a complete installation of Pigsty on the meta node, Pigsty will build the local source directly with that package. Otherwise, Pigsty will download all packages from the network.

The official offline package is made based on CentOS 7.8.2003 OS. Please refer to the FAQ to see all the dependency packages and install them directly from the upstream. Or visit Github on a machine with the same OS, make an offline package, and copy it to a network isolated environment.

You can download the offline package in advance on the meta node where Pigsty is to be installed (just on a meta node to /tmp/pkg.tgz).

curl https://github.com/Vonng/pigsty/releases/download/v1.5.1/pkg.tgz -o /tmp/pkg.tgz   # Github Release，Most authoritative
curl http://download.pigsty.cc/v1.5.1/pkg.tgz -o /tmp/pkg.tgz                      # Download with CDN in China

Also, the download script in the root of the pigsty project can be used to download offline packages.

./download pkg.tgz    # Download the current version of pkg.tgz from Github/CDN to /tmp/pkg.tgz
./download pkg        # Download the current version of pkg.tgz from Github/CDN and extract it to /www/pigsty

Finally, Baidu Web-Disk also provides an offline package for download: https://pan.baidu.com/s/1DZIa9X2jAxx69Zj-aRHoaw?pwd=8su9

Vagrant

Local VM configuration is relatively cumbersome, and Vagrant can solve this problem.

Vagrant makes it fast and straightforward to pull several VMs on a laptop, PC, or Mac, depending on the user’s needs. Users need to express their requirements for VMs in a vagrant configuration file.

We have provided a sample Vagrant configuration file. Vagrantfile

In the Vagrant configuration file, four VMs, including a 2-core/4GB central control/meta node meta and three 1-core/1GB database nodes node-1, node-2, node3.

When using the sandbox via shortcuts like make up, make new, and make demo, only one meta node meta is used by default. And make up4, make new4, and make demo4 use all the VMs. The N value defines the number of additional database nodes. The user can also modify the number of CPU cores and memory resources per machine, etc.

Vagrantfile Example 
IMAGE_NAME = "centos/7"
N=3       # number of extra database nodes, can be 0

Vagrant.configure("2") do |config|
    config.vm.box = IMAGE_NAME
    config.vm.box_check_update = false
    config.ssh.insert_key = false

    # meta (admin) node
    config.vm.define "meta", primary: true do |meta|   # default ssh alias for admin node is `meta`
        meta.vm.hostname = "meta"
        meta.vm.network "private_network", ip: "10.10.10.10"
        meta.vm.provider "virtualbox" do |v|
            v.linked_clone = true
            v.customize [
                    "modifyvm", :id,
                    "--memory", 4096, "--cpus", "2",   # default mem and cpu for meta node: 2C/4GB by default
                    "--nictype1", "virtio", "--nictype2", "virtio",
                    "--hwv·irtex", "on", "--ioapic", "on", "--rtcuseutc", "on", "--vtxvpid", "on", "--largepages", "on"
                ]
        end
        meta.vm.provision "shell", path: "provision.sh"
    end

    # Init N database nodes
    (1..N).each do |i|
        config.vm.define "node-#{i}" do |node|      # default ssh alias for database nodes are `node-{1,2,3}`
            node.vm.box = IMAGE_NAME
            node.vm.network "private_network", ip: "10.10.10.#{i + 10}"
            node.vm.hostname = "node-#{i}"
            node.vm.provider "virtualbox" do |v|
                v.linked_clone = true
                v.customize [
                        "modifyvm", :id,
                        "--memory", 2048, "--cpus", "1",   # resource for database node: 1C/2GB by default
                        "--nictype1", "virtio", "--nictype2", "virtio",
                        "--hwvirtex", "on", "--ioapic", "on", "--rtcuseutc", "on", "--vtxvpid", "on", "--largepages", "on"
                    ]
            end
            node.vm.provision "shell", path: "provision.sh"
        end
    end
end

The vagrant binary will invoke Virtualbox by default to complete the creation of the local VMs defined in the Vagrant configuration file. Go to vagrant in the Pigsty root directory and execute vagrant up to bring up the four VMs. The Makefile provides some wrappers for the original vagrant command.

The default VM image used by the sandbox is IMAGE_NAME = "centos/7".

Virtualbox

Virtualbox is an open-source and free cross-platform VM software. Installing Virtualbox on MacOS is very simple: brew install virtualbox, and is similar on other OS.

After installing Virtualbox, you may need to reboot your computer to load the VM kernel module. Pigsty requires an x86_64 runtime environment, and Macbooks with M1 chips installed may not be able to run Virtualbox properly.

16 - Sandbox

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty supports both local sandbox and cloud sandbox for quickly preparing a standard 1/4 node demo environment locally or in the cloud.

Pigsty provides a sandbox environment. Ultimately creating and running through the sandbox installation and deployment process for deployment in production envs have Pigsty very helpful.

Introduction

The config specifications, object identities, and default database predetermined environment of the sandbox are consistent in both the local and cloud versions.

The sandbox meta node IP is fixed to: 10.10.10.10. 10.10.10.10 is also a placeholder for the IP of the meta node in all config templates, which is used as the actual IP of the meta node when executing config.

You can use a single-node sandbox with a meta, deployed with complete infrastructure, and a single instance Postgres database pg-meta.

  • meta 10.10.10.10 pg-meta.pg-meta-1

The single-node sandbox is suitable for personal development, experimentation, and learning; the four-node sandbox can demonstrate Pigsty’s capabilities, data analysis and visualization, design, demonstration, and distribution of interactive data applications. Please select as you need.

There are three additional nodes in the four-node sandbox, with a set of three-node PostgreSQL cluster pg-test.

  • node-1 10.10.10.11 pg-test.pg-test-1
  • node-2 10.10.10.12 pg-test.pg-test-2
  • node-3 10.10.10.13 pg-test.pg-test-3

Also, the sandbox will use the following two IPs with two static DNS records for accessing the database cluster.

  • 10.10.10.2 pg-meta
  • 10.10.10.2 pg-test

Pigsty offers a local sandbox based on Vagrant (pulling up local VMs using Virtualbox), and a cloud sandbox based on Terraform (creating VMs using the cloud vendor API).

  • Local sandbox can be run on Mac/PC for free. If running a full 4-node sandbox, your Mac/PC should have at least 4C/8G.

  • Cloud sandbox can be easily shown and shared. You need to create a cloud account. VM resources are created and used on-demand and can be destroyed with one click after use, which is also very cheap.

Local Sandbox

The Pigsty local sandbox relies on Vagrant to host the local Virtualbox VM.

Before using Pigsty sandbox, you need to install Vagrant and Virtualbox in your operating system. You can also create VMs for standard installation and deployment by choosing other VM software (Parallel Desktop, VMWare).

Quick Start

Make sure that Vagrant and Virtualbox are installed and available. On macOS, you can use homebrew to install both with one click (requires reboot).

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" # Install homebrew
brew install vagrant virtualbox # Installing Vagrant and Virtualbox on a MacOS host

You can install software dependencies on macOS, configure local static DNS, and pull up VMs with the following four shortcutS. A few additional manual steps are required under Windows and Linux.

make deps    # Install homebrew, and install vagrant and virtualbox via homebrew (requires reboot)
make dns     # Write a static domain name to local /etc/hosts (requires sudo password)
make start   # Pull up a single meta node using Vagrant (4 nodes for start4)

Next, you can ssh meta to log in to the default meta node. SSH sudo for the meta node access to all nodes is already configured, and you can execute the Pigsty installation.

Vagrant

Local VM configuration is relatively cumbersome, and Vagrant can solve this problem.

Vagrant makes it fast and straightforward to pull several VMs on a laptop, PC, or Mac, depending on the user’s needs. Users need to express their requirements for VMs in a vagrant configuration file.

We have provided a sample Vagrant configuration file. Vagrantfile

In the Vagrant configuration file, four VMs, including a 2-core/4GB central control/meta node meta and three 1-core/1GB database nodes node-1, node-2, node3.

When using the sandbox via shortcuts like make up, make new, and make demo, only one meta node meta is used by default. And make up4, make new4, and make demo4 use all the VMs. The N value defines the number of additional database nodes. The user can also modify the number of CPU cores and memory resources per machine, etc.

Vagrantfile Example 
IMAGE_NAME = "centos/7"
N=3  # Number of database machine nodes, can be modified to 0

Vagrant.configure("2") do |config|
    config.vm.box = IMAGE_NAME
    config.vm.box_check_update = false
    config.ssh.insert_key = false

    # Meta Nodes
    config.vm.define "meta", primary: true do |meta|  # The default ssh alias for the meta node is `meta`
        meta.vm.hostname = "meta"
        meta.vm.network "private_network", ip: "10.10.10.10"
        meta.vm.provider "virtualbox" do |v|
            v.linked_clone = true
            v.customize [
                    "modifyvm", :id,
                    "--memory", 4096, "--cpus", "2",   # Memory and CPU cores for meta nodes: default is 2 cores/4GB
                    "--nictype1", "virtio", "--nictype2", "virtio",
                    "--hwv·irtex", "on", "--ioapic", "on", "--rtcuseutc", "on", "--vtxvpid", "on", "--largepages", "on"
                ]
        end
        meta.vm.provision "shell", path: "provision.sh"
    end

    # Initialize N database nodes
    (1..N).each do |i|
        config.vm.define "node-#{i}" do |node|  # The default ssh aliases for the database nodes are `node-{1,2,3}`
            node.vm.box = IMAGE_NAME
            node.vm.network "private_network", ip: "10.10.10.#{i + 10}"
            node.vm.hostname = "node-#{i}"
            node.vm.provider "virtualbox" do |v|
                v.linked_clone = true
                v.customize [
                        "modifyvm", :id,
                        "--memory", 2048, "--cpus", "1", # Database node memory and CPU cores: default is 1 core/2GB
                        "--nictype1", "virtio", "--nictype2", "virtio",
                        "--hwvirtex", "on", "--ioapic", "on", "--rtcuseutc", "on", "--vtxvpid", "on", "--largepages", "on"
                    ]
            end
            node.vm.provision "shell", path: "provision.sh"
        end
    end
end

The vagrant binary will invoke Virtualbox by default to complete the creation of the local VMs defined in the Vagrant configuration file. Go to vagrant in the Pigsty root directory and execute vagrant up to bring up the four VMs. The Makefile provides some wrappers for the original vagrant command.

The default VM image used by the sandbox is IMAGE_NAME = "centos/7".

Virtualbox

Virtualbox is an open-source and free cross-platform VM software. Installing Virtualbox on MacOS is very simple: brew install virtualbox, similar to other OS.

After installing Virtualbox, you may need to reboot your computer to load the VM kernel module. Pigsty requires an x86_64 runtime environment, and Macbooks with M1 chips installed may not be able to run Virtualbox properly.

DNS Config

Pigsty accesses all web systems via domain name by default. If you do not have a DNS server or public domain name, you can use local static DNS records. The static DNS records used by the sandbox are shown below.

# pigsty dns records
10.10.10.10 meta pigsty c.pigsty g.pigsty l.pigsty p.pigsty a.pigsty cli.pigsty lab.pigsty api.pigsty mx.pigsty
10.10.10.11 node-1   # sandbox node node-1
10.10.10.12 node-2   # sandbox node node-2
10.10.10.13 node-3   # sandbox node node-3
10.10.10.2  pg-meta  # sandbox vip for pg-meta
10.10.10.3  pg-test  # sandbox vip for pg-test

On macOS and Linux, running sudo make dns will write the above records to /etc/hosts (requires sudo access). On Windows, you will need to add them manually to C:\Windows\System32\drivers\etc\hosts.

Cloud Sandbox

You can also use cloud VMs that are ready to use and destroy.

Terraform

Terraform is an open-source and free infrastructure or code tool. Just declare the required cloud VMs, network, security group configurations, etc., and pull up the corresponding resources with one click.

To install Terraform under MacOS, execute brew install terraform. You will need a cloud account to obtain AccessKey and AccessSecret credentials.

Config file

The project root dir terraform/ provides Terraform files for several cloud vendors. You can use these templates to quickly request VM resources on the cloud for Pigsty deployment. Here is an example of Ali cloud.

cd terraform        # Go to the terraform dir
vi alicloud.tf      # Edit the config file, fill in your AliCloud AccessKey and SecretKey
AliCloud Sample Terraform 
provider "alicloud" {
  access_key = "xxxxxx"
  secret_key = "xxxxxx"
  region = "cn-beijing"
}

# use 10.10.10.0/24 cidr block as demo network
resource "alicloud_vpc" "vpc" {
  vpc_name   = "pigsty-demo-network"
  cidr_block = "10.10.10.0/24"
}

# add virtual switch for pigsty demo network
resource "alicloud_vswitch" "vsw" {
  vpc_id     = "${alicloud_vpc.vpc.id}"
  cidr_block = "10.10.10.0/24"
  zone_id    = "cn-beijing-k"
}

# add default security group and allow all tcp traffic
resource "alicloud_security_group" "default" {
  name   = "default"
  vpc_id = "${alicloud_vpc.vpc.id}"
}
resource "alicloud_security_group_rule" "allow_all_tcp" {
  ip_protocol       = "tcp"
  type              = "ingress"
  nic_type          = "intranet"
  policy            = "accept"
  port_range        = "1/65535"
  priority          = 1
  security_group_id = "${alicloud_security_group.default.id}"
  cidr_ip           = "0.0.0.0/0"
}

# https://registry.terraform.io/providers/aliyun/alicloud/latest/docs/resources/instance
resource "alicloud_instance" "pg-meta-1" {
  instance_name              = "pg-meta-1"
  host_name                  = "pg-meta-1"
  instance_type              = "ecs.s6-c1m2.small"
  vswitch_id                 = "${alicloud_vswitch.vsw.id}"
  security_groups            = ["${alicloud_security_group.default.id}"]
  image_id                   = "centos_7_8_x64_20G_alibase_20200914.vhd"
  password                   = "PigstyDemo4"
  private_ip                 = "10.10.10.10"
  internet_max_bandwidth_out = 40 # 40Mbps , alloc a public IP
}

resource "alicloud_instance" "pg-test-1" {
  instance_name   = "pg-test-1"
  host_name       = "pg-test-1"
  instance_type   = "ecs.s6-c1m1.small"
  vswitch_id      = "${alicloud_vswitch.vsw.id}"
  security_groups = ["${alicloud_security_group.default.id}"]
  image_id        = "centos_7_8_x64_20G_alibase_20200914.vhd"
  password        = "PigstyDemo4"
  private_ip      = "10.10.10.11"
}

resource "alicloud_instance" "pg-test-2" {
  instance_name   = "pg-test-2"
  host_name       = "pg-test-2"
  instance_type   = "ecs.s6-c1m1.small"
  vswitch_id      = "${alicloud_vswitch.vsw.id}"
  security_groups = ["${alicloud_security_group.default.id}"]
  image_id        = "centos_7_8_x64_20G_alibase_20200914.vhd"
  password        = "PigstyDemo4"
  private_ip      = "10.10.10.12"
}

resource "alicloud_instance" "pg-test-3" {
  instance_name   = "pg-test-3"
  host_name       = "pg-test-3"
  instance_type   = "ecs.s6-c1m1.small"
  vswitch_id      = "${alicloud_vswitch.vsw.id}"
  security_groups = ["${alicloud_security_group.default.id}"]
  image_id        = "centos_7_8_x64_20G_alibase_20200914.vhd"
  password        = "PigstyDemo4"
  private_ip      = "10.10.10.13"
}


output "meta_ip" {
  value = "${alicloud_instance.pg-meta-1.public_ip}"
}

Execution Plan

First, use the terraform command to create the cloud resource defined above (Pay on demand).

terraform init      # Install terraform provider: aliyun (required only for the first time)
terraform apply     # Generate execution plans: create VMs, virtual segments/switches/security groups

After running apply and entering yes, terraform will call AliCloud API to create the corresponding VM resource.

Terraform Execution Results 
Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # alicloud_instance.pg-meta-1 will be created
  + resource "alicloud_instance" "pg-meta-1" {
      + availability_zone                  = (known after apply)
      + credit_specification               = (known after apply)
      + deletion_protection                = false
      + dry_run                            = false
      + host_name                          = "pg-meta-1"
      + id                                 = (known after apply)
      + image_id                           = "centos_7_8_x64_20G_alibase_20200914.vhd"
      + instance_charge_type               = "PostPaid"
      + instance_name                      = "pg-meta-1"
      + instance_type                      = "ecs.s6-c1m2.small"
      + internet_charge_type               = "PayByTraffic"
      + internet_max_bandwidth_in          = (known after apply)
      + internet_max_bandwidth_out         = 40
      + key_name                           = (known after apply)
      + password                           = (sensitive value)
      + private_ip                         = "10.10.10.10"
      + public_ip                          = (known after apply)
      + role_name                          = (known after apply)
      + secondary_private_ip_address_count = (known after apply)
      + secondary_private_ips              = (known after apply)
      + security_groups                    = (known after apply)
      + spot_strategy                      = "NoSpot"
      + status                             = "Running"
      + subnet_id                          = (known after apply)
      + system_disk_category               = "cloud_efficiency"
      + system_disk_performance_level      = (known after apply)
      + system_disk_size                   = 40
      + volume_tags                        = (known after apply)
      + vswitch_id                         = (known after apply)
    }

  # alicloud_instance.pg-test-1 will be created
  + resource "alicloud_instance" "pg-test-1" {
      + availability_zone                  = (known after apply)
      + credit_specification               = (known after apply)
      + deletion_protection                = false
      + dry_run                            = false
      + host_name                          = "pg-test-1"
      + id                                 = (known after apply)
      + image_id                           = "centos_7_8_x64_20G_alibase_20200914.vhd"
      + instance_charge_type               = "PostPaid"
      + instance_name                      = "pg-test-1"
      + instance_type                      = "ecs.s6-c1m1.small"
      + internet_max_bandwidth_in          = (known after apply)
      + internet_max_bandwidth_out         = 0
      + key_name                           = (known after apply)
      + password                           = (sensitive value)
      + private_ip                         = "10.10.10.11"
      + public_ip                          = (known after apply)
      + role_name                          = (known after apply)
      + secondary_private_ip_address_count = (known after apply)
      + secondary_private_ips              = (known after apply)
      + security_groups                    = (known after apply)
      + spot_strategy                      = "NoSpot"
      + status                             = "Running"
      + subnet_id                          = (known after apply)
      + system_disk_category               = "cloud_efficiency"
      + system_disk_performance_level      = (known after apply)
      + system_disk_size                   = 40
      + volume_tags                        = (known after apply)
      + vswitch_id                         = (known after apply)
    }

  # alicloud_instance.pg-test-2 will be created
  + resource "alicloud_instance" "pg-test-2" {
      + availability_zone                  = (known after apply)
      + credit_specification               = (known after apply)
      + deletion_protection                = false
      + dry_run                            = false
      + host_name                          = "pg-test-2"
      + id                                 = (known after apply)
      + image_id                           = "centos_7_8_x64_20G_alibase_20200914.vhd"
      + instance_charge_type               = "PostPaid"
      + instance_name                      = "pg-test-2"
      + instance_type                      = "ecs.s6-c1m1.small"
      + internet_max_bandwidth_in          = (known after apply)
      + internet_max_bandwidth_out         = 0
      + key_name                           = (known after apply)
      + password                           = (sensitive value)
      + private_ip                         = "10.10.10.12"
      + public_ip                          = (known after apply)
      + role_name                          = (known after apply)
      + secondary_private_ip_address_count = (known after apply)
      + secondary_private_ips              = (known after apply)
      + security_groups                    = (known after apply)
      + spot_strategy                      = "NoSpot"
      + status                             = "Running"
      + subnet_id                          = (known after apply)
      + system_disk_category               = "cloud_efficiency"
      + system_disk_performance_level      = (known after apply)
      + system_disk_size                   = 40
      + volume_tags                        = (known after apply)
      + vswitch_id                         = (known after apply)
    }

  # alicloud_instance.pg-test-3 will be created
  + resource "alicloud_instance" "pg-test-3" {
      + availability_zone                  = (known after apply)
      + credit_specification               = (known after apply)
      + deletion_protection                = false
      + dry_run                            = false
      + host_name                          = "pg-test-3"
      + id                                 = (known after apply)
      + image_id                           = "centos_7_8_x64_20G_alibase_20200914.vhd"
      + instance_charge_type               = "PostPaid"
      + instance_name                      = "pg-test-3"
      + instance_type                      = "ecs.s6-c1m1.small"
      + internet_max_bandwidth_in          = (known after apply)
      + internet_max_bandwidth_out         = 0
      + key_name                           = (known after apply)
      + password                           = (sensitive value)
      + private_ip                         = "10.10.10.13"
      + public_ip                          = (known after apply)
      + role_name                          = (known after apply)
      + secondary_private_ip_address_count = (known after apply)
      + secondary_private_ips              = (known after apply)
      + security_groups                    = (known after apply)
      + spot_strategy                      = "NoSpot"
      + status                             = "Running"
      + subnet_id                          = (known after apply)
      + system_disk_category               = "cloud_efficiency"
      + system_disk_performance_level      = (known after apply)
      + system_disk_size                   = 40
      + volume_tags                        = (known after apply)
      + vswitch_id                         = (known after apply)
    }

  # alicloud_security_group.default will be created
  + resource "alicloud_security_group" "default" {
      + id                  = (known after apply)
      + inner_access        = (known after apply)
      + inner_access_policy = (known after apply)
      + name                = "default"
      + security_group_type = "normal"
      + vpc_id              = (known after apply)
    }

  # alicloud_security_group_rule.allow_all_tcp will be created
  + resource "alicloud_security_group_rule" "allow_all_tcp" {
      + cidr_ip           = "0.0.0.0/0"
      + id                = (known after apply)
      + ip_protocol       = "tcp"
      + nic_type          = "intranet"
      + policy            = "accept"
      + port_range        = "1/65535"
      + priority          = 1
      + security_group_id = (known after apply)
      + type              = "ingress"
    }

  # alicloud_vpc.vpc will be created
  + resource "alicloud_vpc" "vpc" {
      + cidr_block        = "10.10.10.0/24"
      + id                = (known after apply)
      + ipv6_cidr_block   = (known after apply)
      + name              = (known after apply)
      + resource_group_id = (known after apply)
      + route_table_id    = (known after apply)
      + router_id         = (known after apply)
      + router_table_id   = (known after apply)
      + status            = (known after apply)
      + vpc_name          = "pigsty-demo-network"
    }

  # alicloud_vswitch.vsw will be created
  + resource "alicloud_vswitch" "vsw" {
      + availability_zone = (known after apply)
      + cidr_block        = "10.10.10.0/24"
      + id                = (known after apply)
      + name              = (known after apply)
      + status            = (known after apply)
      + vpc_id            = (known after apply)
      + vswitch_name      = (known after apply)
      + zone_id           = "cn-beijing-k"
    }

Plan: 8 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + meta_ip = (known after apply)

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

alicloud_vpc.vpc: Creating...
alicloud_vpc.vpc: Creation complete after 6s [id=vpc-2zed78z7n5z06o1dmydhj]
alicloud_security_group.default: Creating...
alicloud_vswitch.vsw: Creating...
alicloud_security_group.default: Creation complete after 1s [id=sg-2ze7x7zu8tcdsefroofa]
alicloud_security_group_rule.allow_all_tcp: Creating...
alicloud_security_group_rule.allow_all_tcp: Creation complete after 0s [id=sg-2ze7x7zu8tcdsefroofa:ingress:tcp:1/65535:intranet:0.0.0.0/0:accept:1]
alicloud_vswitch.vsw: Creation complete after 6s [id=vsw-2zejctjdr16ryz194jxz4]
alicloud_instance.pg-test-3: Creating...
alicloud_instance.pg-test-2: Creating...
alicloud_instance.pg-test-1: Creating...
alicloud_instance.pg-meta-1: Creating...
alicloud_instance.pg-test-3: Still creating... [10s elapsed]
alicloud_instance.pg-test-2: Still creating... [10s elapsed]
alicloud_instance.pg-test-1: Still creating... [10s elapsed]
alicloud_instance.pg-meta-1: Still creating... [10s elapsed]
alicloud_instance.pg-meta-1: Creation complete after 16s [id=i-2zef4frw6kezb47339wr]
alicloud_instance.pg-test-1: Still creating... [20s elapsed]
alicloud_instance.pg-test-2: Still creating... [20s elapsed]
alicloud_instance.pg-test-3: Still creating... [20s elapsed]
alicloud_instance.pg-test-2: Creation complete after 23s [id=i-2zefzvz0fyl7mloc4v30]
alicloud_instance.pg-test-1: Still creating... [30s elapsed]
alicloud_instance.pg-test-3: Still creating... [30s elapsed]
alicloud_instance.pg-test-3: Creation complete after 33s [id=i-2zeeyodo2pc8b1k2d167]
alicloud_instance.pg-test-1: Creation complete after 33s [id=i-2zef4frw6kezb47339ws]

SSH Config and Tweaking

The manager will assign a pay-per-use public IP, which you can also print out using the command terraform output.

# Print public IP and root password
ssh_pass='PigstyDemo4'
public_ip=$(terraform output | grep -Eo '[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}')
echo "meta node: root:${ssh_pass}@${public_ip}"

Next, let’s configure the local SSH login to the cloud manager (default user root, password PigstyDemo4).

# Create ~/.ssh/pigsty_terraform file containing SSH definitions for the cloud manager (optional)
cat > ~/.ssh/pigsty_terraform <<-EOF
Host demo
  User root
  HostName ${public_ip}
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication yes
EOF
chmod 0600 ~/.ssh/pigsty_terraform

# Enable this config
if ! grep --quiet "Include ~/.ssh/pigsty_terraform" ~/.ssh/config ; then
    (echo 'Include ~/.ssh/pigsty_terraform' && cat ~/.ssh/config) >  ~/.ssh/config.tmp;
    mv ~/.ssh/config.tmp ~/.ssh/config && chmod 0600 ~/.ssh/config;
fi

You can access the cloud manager via the SSH alias demo.

# Add local to meta node for password-free access
sshpass -p ${ssh_pass} ssh-copy-id demo

Now, it is possible to access the node from the local password-free. If only a single node installation is required, this will do. Next, complete the standard installation on that meta node.

DNS Config

Pigsty accesses all web systems via domain name by default and does not recommend using IP: Port to access the primary system’s web interface.

The static DNS records used by the cloud sandbox are shown below, and you need to fill in the public IP of the meta node.

<public_ip> meta pigsty c.pigsty g.pigsty l.pigsty p.pigsty a.pigsty cli.pigsty lab.pigsty api.pigsty mx.pigsty

In macOS and Linux, you need to write the above records to /etc/hosts (requires sudo access), and in Windows, you need to add them manually to C:\Windows\System32\drivers\etc\hosts.

Caveat

The AliCloud VM CentOS 7.8 mirror runs with nscd, which locks out the glibc version and causes RPM dependency errors during installation.

Run yum remove -y nscd on all machines to resolve this issue.

17 - Monitoring System

Historical documentation restored from the Pigsty v1.5.1 tag.

How to use Pigsty to monitor existing PostgreSQL instances?

For existing Pigsty instances that Pigsty does not create, some additional configuration is required if they are monitored using Pigsty’s monitoring system.

TL; DR

  1. Create the monitoring object in the target instance: monitoring object configuration.

  2. Declare the cluster in the inventory.

    pg-test:
  hosts:                                # Assign unique local ports to each instance
    10.10.10.11: { pg_seq: 1, pg_role: primary , pg_exporter_port: 20001}
    10.10.10.12: { pg_seq: 2, pg_role: replica , pg_exporter_port: 20002}
    10.10.10.13: { pg_seq: 3, pg_role: offline , pg_exporter_port: 20003}
  vars:
    pg_cluster: pg-test                 # Fill in the cluster name
    pg_version: 14                      # Fill in the major version of the database
    pg_databases: [{ name: test }]      # Fill in the database list (each database object as an array element)

#  Provide monitoring user passwords in global/cluster/instance configs  pg_monitor_username/pg_monitor_password

  3. Execute the playbook against the cluster: ./pgsql-monly.yml -l pg-test.

  4. The playbook registers the target PostgreSQL data source in Grafana so that PGCAT functionality is fully available. The playbook deploys PG Exporter on the meta node to monitor remote PG instances, so pure database-related metrics in PGSQL are available. However, host node, connection pool, load balancing, and HA Patroni metrics are unavailable.

Overview

Suppose you want to use Pigsty for monitoring only. In that case, you can use monly mode. You can use Pigsty to manage and monitor other PostgreSQL instances (currently, 10+ versions are supported by default, the pg_exporter configuration file can manually modify older versions).

First, you need to complete the standard installation process of Pigsty on one meta node, and then you can connect more database instances to monitoring. According to the access rights of the target database node, there are two different cases.

Target nodes can be managed.

If the target DB node can be managed by Pigsty (ssh reachable, sudo available), you can use the pg-exporter task of the pgsql.yml playbook to deploy the monitoring component on the target node in the same way: PG Exporter. You can also use the other tasks of the playbook to deploy additional components and their monitoring on the existing instance node: connection pool Pgbouncer and load balancer HAProxy. In addition, the node-exporter and promtail tasks in nodes.yml can also be used to deploy the host node monitoring and log collection components on existing instance nodes. And log collection components.

Since the target database cluster already exists, you will need to manually create monitoring users, modes, and extensions on the target database cluster as described in this section. The rest of the process is no different from full deployment.

# Modify the pigsty config, add the yum repo to the node, and install the package via yum
exporter_install: yum # none|yum|binary, none by default
exporter_repo_url: http://<your primary ip address>/pigsty.repo

./nodes.yml -l <yourcluster> -t node-exporter  # Deployment Node Metrics Monitoring
./nodes.yml -l <yourcluster> -t promtail       # Deployment Node Log Collection
./pgsql.yml -l <yourcluster> -t pg-exporter    # Deployment of PG metrics monitoring collection

Database connection string only.

If you can only access the target database using a PGURL (database connection string), consider monitoring the target database’s Monly mode/Basic mode. All monitoring components are deployed on the meta node where Pigsty is installed in this mode. The monitoring system will not have metrics related to nodes, connection pools, load balancers, and HA components, but the database and real-time status information in the Data Catalog will still be available.

To perform a lean monitoring deployment, you will also need to manually create monitoring users, modes, and extensions on the target database cluster as described in this section and ensure that the target database can be accessed from the meta node using monitoring users. After that, execute the pgsql-monly.yml playbook against the target cluster to complete the deployment.

This article focuses on this monly deployment mode.

Figure: the monly mode architecture, with multiple PG Exporter, deployed locally on the management machine for monitoring multiple remote database instances.

Difference

The Pigsty monitoring system consists of three core modules:

Matter \ Level L1 L2 L3
Name Basic Deployment Managed Deployment Managed Deployment
English basic managed full
Scenes Only the connection string DB already exists, nodes can be managed Instances created by Pigsty
PGCAT Function ✅ Full Availability ✅ Full Availability ✅ Full Availability
PGSQL Function ✅ Limited PG metrics ✅ Limited PG and node metrics ✅ Full Function
Connection Pool Metrics ❌ Not available ⚠️ Optional ✅ Pre-installed
Load Balancer Metrics ❌ Not available ⚠️ Optional ✅ Pre-installed
PGLOG Function ❌ Not available ⚠️ Optional ⚠️ Optional
PG Exporter ⚠️ Deployed on meta nodes ✅ Deployed on DB nodes ✅ Deployed on DB nodes
Node Exporter ❌ Non-deployment ✅ Deployed on DB nodes ✅ Deployed on DB nodes
Intrusion into DB nodes ✅ Non-intrusive ⚠️ Installing Exporter ⚠️ Fully managed by Pigsty
Monitor existing instances ✅ Can support ✅ Can support ❌ For Pigsty-hosted instances only
Monitoring users and views Manually created Manually created Pigsty automatically creates
Deployment Usage Playbook pgsql-monly.yml pgsql.yml -t pg-exporter
nodes.yml -t node-exporter		 pgsql.yml -t pg-exporter
nodes.yml -t node-exporter
Required Privileges PGURLs reachable by meta nodes DB node ssh and sudo privileges DB node ssh and sudo privileges
Function Overview Basic functions:PGCAT+PGSQL Most functions Full Functions

Basic Deploy

Deploying a monitoring system for a database instance is divided into three steps: Prepare Targets, Modify Inventory, and Execute Playbook.

Prepare Targets

To include an external existing PostgreSQL instance in monitoring, a connection string that can be used to access the instance/cluster is required. It is recommended to use a dedicated monitoring user to avoid privilege leaks.

  • Monitor User: The default user name used is dbuser_monitor, which should belong to the pg_monitor role group or ensure that it has access to the relevant view.
  • Monitor Auth: Default is password access. You need to ensure the HBA policy allows the monitor user to access the database locally from the management machine or DB node.
  • Monitor Mode: Fixed using the name monitor for installing additional monitor views with extended plugins, optional but highly recommended.
  • Monitor Extensions: It is highly recommended to enable the monitor extension pg_stat_statements with PG.

For details on the preparation of monitoring objects, please refer to Monitoring Object Config.

Modify Inventory

This target cluster must be declared in the inventory (config file or CMDB). For example, specify the identity for the cluster with the instance. It is also necessary to manually assign a unique local port ( pg_exporter_port) to each instance at the instance level.

The following is a sample database cluster declaration.

pg-test:
  hosts:                                # Assign unique local ports to each instance
    10.10.10.11: { pg_seq: 1, pg_role: primary , pg_exporter_port: 20001}
    10.10.10.12: { pg_seq: 2, pg_role: replica , pg_exporter_port: 20002}
    10.10.10.13: { pg_seq: 3, pg_role: offline , pg_exporter_port: 20003}
  vars:
    pg_cluster: pg-test                 # Fill in cluster name
    pg_version: 14                      # Fill in the major version of the database
    pg_databases: [{ name: test }]      # Fill in the database list (each database object as an array element)

#  Provide monitoring user passwords in global/cluster/instance configs  pg_monitor_username/pg_monitor_password

Even if you access the database through a domain name, you still need to declare the database cluster by filling in the actual IP.

To enable the PGCAT feature, you need to explicitly list in pg_databases the list of database names of the target cluster registered as Grafana’s data source and can access the Catalog data of that instance through Grafana. If you do not use the PGCAT function, do not set this variable or set it to an empty array.

Connect Info

Note: Pigsty will default generate the monitor connection string using the following rules. However, the pg_exporter_url will directly override the spliced connection string if it exists.

postgres://{{ pg_monitor_username }}:{{ pg_monitor_password }}@{{ inventory_hostname }}:{{ pg_port }}/postgres?sslmode=disable

You can constantly monitor user/password globally or configure the following connection parameters on-demand at the cluster or instance level.

pg_monitor_username: dbuser_monitor  # Monitor user name, no need to configure here if using the global config
pg_monitor_password: DBUser.Monitor  # Monitor user passwords, no need to configure here if using the global config
pg_port: 5432                        # If you use a non-standard database port, modify it here
Example: Specifying connect information at the instance-level 
pg-test:
  hosts:                                # Specify the access URL for the instance
    10.10.10.11:
      pg_seq: 1
      pg_role: primary
      pg_exporter_port: 20001
      pg_monitor_username: monitor_user1
      pg_monitor_password: monitor_pass1
    10.10.10.12:
      pg_seq: 2
      pg_role: replica
      pg_exporter_port: 20002           # Specify pg_exporter_url directly
      pg_exporter_url: 'postgres://someuser:[email protected]:5432/postgres?sslmode=disable''
    10.10.10.13:
      pg_seq: 3
      pg_role: offline
      pg_exporter_port: 20003
      pg_monitor_username: monitor_user3
      pg_monitor_password: monitor_pass3
  vars:
    pg_cluster: pg-test                 # Fill in cluster name
    pg_version: 14                      # Fill in the major version of the database
    pg_databases: [{ name: test }]      # Fill in the database list (each database object as an array element)

Execute Playbook

Once the cluster declaration is complete, use the playbook pgsql-monly.yml on the meta node against the target cluster.

./pgsql-monly.yml -l <cluster>     # Complete monitoring deployment on a specified cluster

Monitor Preparation

In monitor mode, the target DB nodes can be managed by Pigsty (ssh reachable, sudo available), and the user will install the following monitoring components on the existing nodes: promtail, node_exporter, pg_exporter.

You can use the node-exporter task in nodes.yml and the pg-exporter task in the pgsql.yml playbook to deploy the monitoring component on the target node: node_exporter with pg_exporter.

Because the target database cluster already exists, you need to create monitor users, modes & extensions on the target database cluster.

# Modify the pigsty config parameters, add the yum repo to the node, and install the package via yum
exporter_install: yum # none|yum|binary, none by default
exporter_repo_url: http://<your primary ip address>/pigsty.repo

./nodes.yml -l <yourcluster> -t promtail       # Deployment node log collection (optional)
./nodes.yml -l <yourcluster> -t node-exporter  # Deployment Node Metrics Monitoring
./pgsql.yml -l <yourcluster> -t pg-exporter    # Deployment of PG metrics collection

When the value of exporter_install is yum, Pigsty will download from the URL specified by the exporter_repo_url Repo file to the node’s local /etc/yum.repos.d. You should fill in the Pigsty local source address on the meta node, for example, http://10.10.10.10/pigsty.repo.

Targets Config

How to configure users, modes, extensions, views, and functions required for monitoring existing instances.

Monitor user

Take the monitoring user dbuser_monitor used by Pigsty by default as an example, and create the following user in the target cluster.

CREATE USER dbuser_monitor;
GRANT pg_monitor TO dbuser_monitor;
COMMENT ON ROLE dbuser_monitor IS 'system monitor user';
ALTER USER dbuser_monitor SET log_min_duration_statement = 1000;
ALTER USER dbuser_monitor PASSWORD 'DBUser.Password'; -- Change monitor user password as needed

Please note that the monitor user and password created need to be the same as pg_monitor_username and pg_monitor_password stay consistent.

Configure the database file pg_hba.conf by adding the following rules to allow monitoring users to access the database from local, and management machines using passwords.

# allow local role monitor with password
local all dbuser_monitor md5
host all dbuser_monitor 127.0.0.1/32 md5
host all dbuser_monitor <management machine IP address>/32 md5

Monitor mode

It is recommended to create a monitor mode and enable at least the pg_stat_statements that come with PG, which provides essential data on query performance. Note: This extension must be included in the database parameter shared_preload_libraries. Modifying this parameter requires a database restart.

Creating an extension mode.

CREATE SCHEMA IF NOT EXISTS monitor; -- Create a monitor-specific mode
GRANT USAGE ON SCHEMA monitor TO dbuser_monitor; -- Allow monitor users to use

Monitor extensions

Create extension plugins.

-- It is highly recommended to enable the pg_stat_statements extension
CREATE EXTENSION IF NOT EXISTS "pg_stat_statements" WITH SCHEMA "monitor";

-- optional other extensions
CREATE EXTENSION IF NOT EXISTS "pgstattuple" WITH SCHEMA "monitor";
CREATE EXTENSION IF NOT EXISTS "pg_qualstats" WITH SCHEMA "monitor";
CREATE EXTENSION IF NOT EXISTS "pg_buffercache" WITH SCHEMA "monitor";
CREATE EXTENSION IF NOT EXISTS "pageinspect" WITH SCHEMA "monitor";
CREATE EXTENSION IF NOT EXISTS "pg_prewarm" WITH SCHEMA "monitor";
CREATE EXTENSION IF NOT EXISTS "pg_visibility" WITH SCHEMA "monitor";
CREATE EXTENSION IF NOT EXISTS "pg_freespacemap" WITH SCHEMA "monitor";

Monitor Views

The monitoring view shows common preprocessing results and wraps privileges for specific monitoring metrics (e.g., shared memory allocation). It is highly recommended to create it in all databases that need to be monitored.

Monitor Views 
--==================================================================--
--                            Monitor Schema                        --
--==================================================================--

----------------------------------------------------------------------
-- cleanse
----------------------------------------------------------------------
CREATE SCHEMA IF NOT EXISTS monitor;
GRANT USAGE ON SCHEMA monitor TO dbuser_monitor;
GRANT USAGE ON SCHEMA monitor TO "{{ pg_admin_username }}";
GRANT USAGE ON SCHEMA monitor TO "{{ pg_replication_username }}";

--==================================================================--
--                            Monitor Views                         --
--==================================================================--

----------------------------------------------------------------------
-- Table bloat estimate : monitor.pg_table_bloat
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_table_bloat CASCADE;
CREATE OR REPLACE VIEW monitor.pg_table_bloat AS
SELECT CURRENT_CATALOG AS datname, nspname, relname , tblid , bs * tblpages AS size,
       CASE WHEN tblpages - est_tblpages_ff > 0 THEN (tblpages - est_tblpages_ff)/tblpages::FLOAT ELSE 0 END AS ratio
FROM (
         SELECT ceil( reltuples / ( (bs-page_hdr)*fillfactor/(tpl_size*100) ) ) + ceil( toasttuples / 4 ) AS est_tblpages_ff,
                tblpages, fillfactor, bs, tblid, nspname, relname, is_na
         FROM (
                  SELECT
                      ( 4 + tpl_hdr_size + tpl_data_size + (2 * ma)
                          - CASE WHEN tpl_hdr_size % ma = 0 THEN ma ELSE tpl_hdr_size % ma END
                          - CASE WHEN ceil(tpl_data_size)::INT % ma = 0 THEN ma ELSE ceil(tpl_data_size)::INT % ma END
                          ) AS tpl_size, (heappages + toastpages) AS tblpages, heappages,
                      toastpages, reltuples, toasttuples, bs, page_hdr, tblid, nspname, relname, fillfactor, is_na
                  FROM (
                           SELECT
                               tbl.oid AS tblid, ns.nspname , tbl.relname, tbl.reltuples,
                               tbl.relpages AS heappages, coalesce(toast.relpages, 0) AS toastpages,
                               coalesce(toast.reltuples, 0) AS toasttuples,
                               coalesce(substring(array_to_string(tbl.reloptions, ' ') FROM 'fillfactor=([0-9]+)')::smallint, 100) AS fillfactor,
                               current_setting('block_size')::numeric AS bs,
                               CASE WHEN version()~'mingw32' OR version()~'64-bit|x86_64|ppc64|ia64|amd64' THEN 8 ELSE 4 END AS ma,
                               24 AS page_hdr,
                               23 + CASE WHEN MAX(coalesce(s.null_frac,0)) > 0 THEN ( 7 + count(s.attname) ) / 8 ELSE 0::int END
                                   + CASE WHEN bool_or(att.attname = 'oid' and att.attnum < 0) THEN 4 ELSE 0 END AS tpl_hdr_size,
                               sum( (1-coalesce(s.null_frac, 0)) * coalesce(s.avg_width, 0) ) AS tpl_data_size,
                               bool_or(att.atttypid = 'pg_catalog.name'::regtype)
                                   OR sum(CASE WHEN att.attnum > 0 THEN 1 ELSE 0 END) <> count(s.attname) AS is_na
                           FROM pg_attribute AS att
                                    JOIN pg_class AS tbl ON att.attrelid = tbl.oid
                                    JOIN pg_namespace AS ns ON ns.oid = tbl.relnamespace
                                    LEFT JOIN pg_stats AS s ON s.schemaname=ns.nspname AND s.tablename = tbl.relname AND s.inherited=false AND s.attname=att.attname
                                    LEFT JOIN pg_class AS toast ON tbl.reltoastrelid = toast.oid
                           WHERE NOT att.attisdropped AND tbl.relkind = 'r' AND nspname NOT IN ('pg_catalog','information_schema')
                           GROUP BY 1,2,3,4,5,6,7,8,9,10
                       ) AS s
              ) AS s2
     ) AS s3
WHERE NOT is_na;
COMMENT ON VIEW monitor.pg_table_bloat IS 'postgres table bloat estimate';

----------------------------------------------------------------------
-- Index bloat estimate : monitor.pg_index_bloat
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_index_bloat CASCADE;
CREATE OR REPLACE VIEW monitor.pg_index_bloat AS
SELECT CURRENT_CATALOG AS datname, nspname, idxname AS relname, tblid, idxid, relpages::BIGINT * bs AS size,
       COALESCE((relpages - ( reltuples * (6 + ma - (CASE WHEN index_tuple_hdr % ma = 0 THEN ma ELSE index_tuple_hdr % ma END)
                                               + nulldatawidth + ma - (CASE WHEN nulldatawidth % ma = 0 THEN ma ELSE nulldatawidth % ma END))
                                  / (bs - pagehdr)::FLOAT  + 1 )), 0) / relpages::FLOAT AS ratio
FROM (
         SELECT nspname,idxname,indrelid AS tblid,indexrelid AS idxid,
                reltuples,relpages,
                current_setting('block_size')::INTEGER                                                               AS bs,
                (CASE WHEN version() ~ 'mingw32' OR version() ~ '64-bit|x86_64|ppc64|ia64|amd64' THEN 8 ELSE 4 END)  AS ma,
                24                                                                                                   AS pagehdr,
                (CASE WHEN max(COALESCE(pg_stats.null_frac, 0)) = 0 THEN 2 ELSE 6 END)                               AS index_tuple_hdr,
                sum((1.0 - COALESCE(pg_stats.null_frac, 0.0)) *
                    COALESCE(pg_stats.avg_width, 1024))::INTEGER                                                     AS nulldatawidth
         FROM pg_attribute
                  JOIN (
             SELECT pg_namespace.nspname,
                    ic.relname                                                   AS idxname,
                    ic.reltuples,
                    ic.relpages,
                    pg_index.indrelid,
                    pg_index.indexrelid,
                    tc.relname                                                   AS tablename,
                    regexp_split_to_table(pg_index.indkey::TEXT, ' ') :: INTEGER AS attnum,
                    pg_index.indexrelid                                          AS index_oid
             FROM pg_index
                      JOIN pg_class ic ON pg_index.indexrelid = ic.oid
                      JOIN pg_class tc ON pg_index.indrelid = tc.oid
                      JOIN pg_namespace ON pg_namespace.oid = ic.relnamespace
                      JOIN pg_am ON ic.relam = pg_am.oid
             WHERE pg_am.amname = 'btree' AND ic.relpages > 0 AND nspname NOT IN ('pg_catalog', 'information_schema')
         ) ind_atts ON pg_attribute.attrelid = ind_atts.indexrelid AND pg_attribute.attnum = ind_atts.attnum
                  JOIN pg_stats ON pg_stats.schemaname = ind_atts.nspname
             AND ((pg_stats.tablename = ind_atts.tablename AND pg_stats.attname = pg_get_indexdef(pg_attribute.attrelid, pg_attribute.attnum, TRUE))
                 OR (pg_stats.tablename = ind_atts.idxname AND pg_stats.attname = pg_attribute.attname))
         WHERE pg_attribute.attnum > 0
         GROUP BY 1, 2, 3, 4, 5, 6
     ) est;
COMMENT ON VIEW monitor.pg_index_bloat IS 'postgres index bloat estimate (btree-only)';


----------------------------------------------------------------------
-- Relation Bloat : monitor.pg_bloat
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_bloat CASCADE;
CREATE OR REPLACE VIEW monitor.pg_bloat AS
SELECT coalesce(ib.datname, tb.datname)                                                   AS datname,
       coalesce(ib.nspname, tb.nspname)                                                   AS nspname,
       coalesce(ib.tblid, tb.tblid)                                                       AS tblid,
       coalesce(tb.nspname || '.' || tb.relname, ib.nspname || '.' || ib.tblid::RegClass) AS tblname,
       tb.size                                                                            AS tbl_size,
       CASE WHEN tb.ratio < 0 THEN 0 ELSE round(tb.ratio::NUMERIC, 6) END                 AS tbl_ratio,
       (tb.size * (CASE WHEN tb.ratio < 0 THEN 0 ELSE tb.ratio::NUMERIC END)) ::BIGINT    AS tbl_wasted,
       ib.idxid,
       ib.nspname || '.' || ib.relname                                                    AS idxname,
       ib.size                                                                            AS idx_size,
       CASE WHEN ib.ratio < 0 THEN 0 ELSE round(ib.ratio::NUMERIC, 5) END                 AS idx_ratio,
       (ib.size * (CASE WHEN ib.ratio < 0 THEN 0 ELSE ib.ratio::NUMERIC END)) ::BIGINT    AS idx_wasted
FROM monitor.pg_index_bloat ib
         FULL OUTER JOIN monitor.pg_table_bloat tb ON ib.tblid = tb.tblid;

COMMENT ON VIEW monitor.pg_bloat IS 'postgres relation bloat detail';


----------------------------------------------------------------------
-- monitor.pg_index_bloat_human
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_index_bloat_human CASCADE;
CREATE OR REPLACE VIEW monitor.pg_index_bloat_human AS
SELECT idxname                            AS name,
       tblname,
       idx_wasted                         AS wasted,
       pg_size_pretty(idx_size)           AS idx_size,
       round(100 * idx_ratio::NUMERIC, 2) AS idx_ratio,
       pg_size_pretty(idx_wasted)         AS idx_wasted,
       pg_size_pretty(tbl_size)           AS tbl_size,
       round(100 * tbl_ratio::NUMERIC, 2) AS tbl_ratio,
       pg_size_pretty(tbl_wasted)         AS tbl_wasted
FROM monitor.pg_bloat
WHERE idxname IS NOT NULL;
COMMENT ON VIEW monitor.pg_index_bloat_human IS 'postgres index bloat info in human-readable format';

----------------------------------------------------------------------
-- monitor.pg_table_bloat_human
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_table_bloat_human CASCADE;
CREATE OR REPLACE VIEW monitor.pg_table_bloat_human AS
SELECT tblname                                          AS name,
       idx_wasted + tbl_wasted                          AS wasted,
       pg_size_pretty(idx_wasted + tbl_wasted)          AS all_wasted,
       pg_size_pretty(tbl_wasted)                       AS tbl_wasted,
       pg_size_pretty(tbl_size)                         AS tbl_size,
       tbl_ratio,
       pg_size_pretty(idx_wasted)                       AS idx_wasted,
       pg_size_pretty(idx_size)                         AS idx_size,
       round(idx_wasted::NUMERIC * 100.0 / idx_size, 2) AS idx_ratio
FROM (SELECT datname,
             nspname,
             tblname,
             coalesce(max(tbl_wasted), 0)                         AS tbl_wasted,
             coalesce(max(tbl_size), 1)                           AS tbl_size,
             round(100 * coalesce(max(tbl_ratio), 0)::NUMERIC, 2) AS tbl_ratio,
             coalesce(sum(idx_wasted), 0)                         AS idx_wasted,
             coalesce(sum(idx_size), 1)                           AS idx_size
      FROM monitor.pg_bloat
      WHERE tblname IS NOT NULL
      GROUP BY 1, 2, 3
     ) d;
COMMENT ON VIEW monitor.pg_table_bloat_human IS 'postgres table bloat info in human-readable format';



----------------------------------------------------------------------
-- Activity Overview: monitor.pg_session
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_session CASCADE;
CREATE OR REPLACE VIEW monitor.pg_session AS
SELECT coalesce(datname, 'all') AS datname, numbackends, active, idle, ixact, max_duration, max_tx_duration, max_conn_duration
FROM (
         SELECT datname,
                count(*)                                         AS numbackends,
                count(*) FILTER ( WHERE state = 'active' )       AS active,
                count(*) FILTER ( WHERE state = 'idle' )         AS idle,
                count(*) FILTER ( WHERE state = 'idle in transaction'
                    OR state = 'idle in transaction (aborted)' ) AS ixact,
                max(extract(epoch from now() - state_change))
                FILTER ( WHERE state = 'active' )                AS max_duration,
                max(extract(epoch from now() - xact_start))      AS max_tx_duration,
                max(extract(epoch from now() - backend_start))   AS max_conn_duration
         FROM pg_stat_activity
         WHERE backend_type = 'client backend'
           AND pid <> pg_backend_pid()
         GROUP BY ROLLUP (1)
         ORDER BY 1 NULLS FIRST
     ) t;
COMMENT ON VIEW monitor.pg_session IS 'postgres activity group by session';


----------------------------------------------------------------------
-- Sequential Scan: monitor.pg_seq_scan
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_seq_scan CASCADE;
CREATE OR REPLACE VIEW monitor.pg_seq_scan AS
    SELECT schemaname                                                        AS nspname,
           relname,
           seq_scan,
           seq_tup_read,
           seq_tup_read / seq_scan                                           AS seq_tup_avg,
           idx_scan,
           n_live_tup + n_dead_tup                                           AS tuples,
           round(n_live_tup * 100.0::NUMERIC / (n_live_tup + n_dead_tup), 2) AS live_ratio
    FROM pg_stat_user_tables
    WHERE seq_scan > 0
      and (n_live_tup + n_dead_tup) > 0
    ORDER BY seq_scan DESC;
COMMENT ON VIEW monitor.pg_seq_scan IS 'table that have seq scan';
Functions to view shared memory allocations (available for PG13 and above) 
DROP FUNCTION IF EXISTS monitor.pg_shmem() CASCADE;
CREATE OR REPLACE FUNCTION monitor.pg_shmem() RETURNS SETOF
    pg_shmem_allocations AS $$ SELECT * FROM pg_shmem_allocations;$$ LANGUAGE SQL SECURITY DEFINER;
COMMENT ON FUNCTION monitor.pg_shmem() IS 'security wrapper for pg_shmem';

18 - PostgreSQL Deployment

Historical documentation restored from the Pigsty v1.5.1 tag.

This article describes several ways to deploy a PostgreSQL cluster using Pigsty: PGSQL-related playbook and config. Please refer to the related doc.

  • Identity Parameters: Introduces the identity parameters required to define a standard PostgreSQL HA cluster.
  • Singleton Deployment: Defines a single instance PostgreSQL cluster.
  • Primary-Replica Cluster: Defines a standard availability cluster with one primary & one replica.
  • Sync-Standby: Define a highly consistent cluster with sync standby and RPO = 0.
  • Quorum Commit: Defines a cluster with higher data consistency: most replicas return commits on the successful side.
  • Offline Replica: Dedicated instances for hosting OLAP analysis, ETL, and interactive personal queries individually.
  • Standby Cluster: Produces real-time online clones of existing clusters for offsite disaster recovery or delayed.
  • Delayed Cluster: For responding to software/human failures such as mistaken table and database deletion, faster than PITR.
  • Cascade Instance: Used to build cascade within a cluster for many replica scenarios (20+) to reduce primary pressure.
  • Citus Deployment: Deploy Citus distributed database cluster.
  • MatrixDB Deployment: Deploy Greenplum7/PostgreSQL12 compatible chronological data warehouse.

Identity

The Core Identity Parameters are information that must be provided when defining a PostgreSQL cluster.

Name Attribute Description Example
pg_cluster MUST, cluster level Cluster name pg-test
pg_role MUST, instance level Instance Role primary, replica
pg_seq MUST, instance level Instance number 1, 2, 3,...

The content of the identity parameter follows the entity naming pattern. Where pg_cluster, pg_role, and pg_seq belong to the core identity parameters, the minimum set of mandatory parameters required to define the database cluster and core identity parameters must be explicitly specified.

  • pg_cluster identities the name of the cluster configured at the cluster level and serves as the top-level namespace for cluster resources.

  • pg_role identities the role of the instance in the cluster, configured at the instance level, with optional values including:

    • primary: the only primary in the cluster, that provides writing services.
    • replica: the ordinary replica in the cluster, takes regular production read-only traffic.
    • offline: an offline replica in the cluster, takes ETL/SAGA/personal user/interactive/analytical queries.
    • standby: a standby replica in the cluster, with synchronous replication and no replication latency (reserved).
    • delayed: a delayed replica in the cluster, explicitly specifying replication delay, used to perform backtracking queries and data salvage (reserved).

  • pg_seq is used to identify the instance within the cluster. Usually, an integer incrementing from 0 or 1 will not be changed once assigned.

  • pg_shard is used to identify the upper-level shard cluster to which the cluster belongs, and only needs to be set if the cluster belongs to a horizontal sharding cluster.

  • pg_sindex is used to identify the cluster’s slice cluster number and only needs to be set if the cluster belongs to a horizontal sharding cluster.

  • pg_instance is the derived identity parameter that uniquely identifies a database instance, with the following composition rules

    {{ pg_cluster }}-{{ pg_seq }}. Since pg_seq is unique within the cluster, this identity is globally unique.

Sharding Cluster

pg_shard and pg_sindex define particular sharded clusters and are optional, currently reserved for Citus and Greenplum.

Suppose a user has a horizontal sharding sharded database cluster with the name test. This cluster consists of four separate clusters: pg-test1, pg-test2, pg-test3, and pg-test-4. The user can bind the identity of pg_shard: test to each database cluster and pg_sindex: 1|2|3|4 to each database cluster separately.

pg-test1:
  vars: {pg_cluster: pg-test1, pg_shard: test, pg_sindex: 1}
  hosts: {10.10.10.10: {pg_seq: 1, pg_role: primary}}
pg-test2:
  vars: {pg_cluster: pg-test1, pg_shard: test, pg_sindex: 2}
  hosts: {10.10.10.11: {pg_seq: 1, pg_role: primary}}
pg-test3:
  vars: {pg_cluster: pg-test1, pg_shard: test, pg_sindex: 3}
  hosts: {10.10.10.12: {pg_seq: 1, pg_role: primary}}
pg-test4:
  vars: {pg_cluster: pg-test1, pg_shard: test, pg_sindex: 4}
  hosts: {10.10.10.13: {pg_seq: 1, pg_role: primary}}

With this definition, you can easily observe the cross-sectional metrics comparison of four horizontal sharding clusters from the PGSQL Shard monitoring dashboard. The same functionality works for Citus and MatrixDB clusters as well.

Singleton

Let’s start with the simplest case.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-test

Use the following command to create a primary database instance on the 10.10.10.11 node.

bin/createpg pg-test

M-S Replication

Pigsty natively supports M-S replication, e.g., to declare a typical one primary & one replica HA database cluster.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-test

Use bin/createpg pg-test to create the cluster. If you have already finished deploying 10.10.10.11 in step 1 singleton deployment, you can also use bin/createpg 10.10.10.12 to expand the cluster.

Sync Standby

Under normal circumstances, PostgreSQL’s replication latency is a few tens of KB/10ms, which is negligible for regular business.

When the primary fails, data that has not yet completed replication will be lost! Replication latency can be a problem when dealing with critical and sophisticated business queries. Or, in a replica, immediately read-your-write after the primary writes, which can also be very sensitive to replication latency.

Sync standbys can solve such problems. A simple way to configure a sync standby is to use the pg_conf = crit template, which automatically enables synchronous replication.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica }
  vars:
    pg_cluster: pg-test
    pg_conf: crit.yml

After the cluster is created, you can also execute pg edit-config <cluster.name> on the meta node, edit the cluster configuration file, change the value of the synchronous_mode to true and apply it.

$ pg edit-config pg-test
---
+++
-synchronous_mode: false
+synchronous_mode: true
 synchronous_mode_strict: false

Apply these changes? [y/N]: y

Quorum Commit

By default, synchronous replication picks an instance from all candidate replicas as a sync standby. Any primary transaction is only considered successfully committed and returned when replicated to the replica and flushed to the disk. A quorum commit can be used if more persistent data is expected. For example, in a 1primary & 3 replicas cluster, at least two replicas successfully flush to disk before a commit is confirmed.

When using quorum commit, you need to modify the synchronous_standby_names in PostgreSQL and the value of synchronous_node_count in Patroni. Assuming that the three replicas are pg-test-2, pg-test-3, and pg-test-4, the following should be configured.

  • synchronous_standby_names = ANY 2 (pg-test-2, pg-test-3, pg-test-4)
  • synchronous_node_count : 2
pg-test:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary } # pg-test-1
    10.10.10.11: { pg_seq: 2, pg_role: replica } # pg-test-2
    10.10.10.12: { pg_seq: 3, pg_role: replica } # pg-test-3
    10.10.10.13: { pg_seq: 4, pg_role: replica } # pg-test-4
  vars:
    pg_cluster: pg-test

Execute pg edit-config pg-test and modify the config as follows.

$ pg edit-config pg-test
---
+++
@@ -82,10 +82,12 @@
     work_mem: 4MB
+    synchronous_standby_names: 'ANY 2 (pg-test-2, pg-test-3, pg-test-4)'

-synchronous_mode: false
+synchronous_mode: true
+synchronous_node_count: 2
 synchronous_mode_strict: false

Apply these changes? [y/N]: y

After the application, the configuration takes effect, and two Sync Standby appear. When the cluster has Failover or expansion and contraction, please adjust these parameters to avoid service unavailability.

+ Cluster: pg-test (7080814403632534854) +---------+----+-----------+-----------------+
| Member    | Host        | Role         | State   | TL | Lag in MB | Tags            |
+-----------+-------------+--------------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.10 | Leader       | running |  1 |           | clonefrom: true |
| pg-test-2 | 10.10.10.11 | Sync Standby | running |  1 |         0 | clonefrom: true |
| pg-test-3 | 10.10.10.12 | Sync Standby | running |  1 |         0 | clonefrom: true |
| pg-test-4 | 10.10.10.13 | Replica      | running |  1 |         0 | clonefrom: true |
+-----------+-------------+--------------+---------+----+-----------+-----------------+

Offline Replica

Data analysis/ETL/personal interactive queries should be placed on the offline replica when the high online business request load.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 2, pg_role: offline } #Define a new offline instance
  vars:
    pg_cluster: pg-test

Use bin/createpg pg-test to create the cluster. If you have already completed singleton deployment and primary-replica-cluster, you can use bin/createpg 10.10.10.13 to expand the cluster and add an offline replica to the cluster.

Offline replicas do not host the replica service by default, and the offline instance will only host read-only traffic if all instances in the replica service are unavailable. If you have only one primary & one replica, or only one primary, you can set the pg_offline_query flag for an offline instance that also hosts the offline service to be used as a quasi-offline instance.

Standby Cluster

You can make a clone of an existing cluster using the Standby Cluster method, which allows for a smooth migration from a current database to a Pigsty cluster.

Just make sure that the pg_upstream parameter is configured on the primary of the backup cluster to pull backups from the original upstream automatically.

# pg-test is the original database
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-test
    pg_version: 14


# Pg-test2 will be the standby cluster of pg-test1.
pg-test2:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary , pg_upstream: 10.10.10.11 } # The actual role is Standby Leader
    10.10.10.13: { pg_seq: 2, pg_role: replica }
  vars:
    pg_cluster: pg-test2
    pg_version: 14          # When making a Standby Cluster, the database major version must be consistent!
bin/createpg pg-test     # Creating the original cluster
bin/createpg pg-test2    # Creating a Backup Cluster

Promote Standby Cluster

When you want to promote the standby cluster to a standalone cluster, edit the Patroni configuration file of the new cluster to remove all standby_cluster configurations, and the Standby Leader in the standby cluster will be elevated to a standalone primary.

pg edit-config pg-test2  # Remove the standby_cluster config definition and apply

Remove the following config: the entire standby_cluster definition section.

-standby_cluster:
-  create_replica_methods:
-  - basebackup
-  host: 10.10.10.11
-  port: 5432

Change Replication Upstream

When a Failover primary change occurs in the source cluster, you need to adjust the replication source of the standby cluster. Execute pg edit-config <cluster> and change the source address in standby_cluster to the new primary, and the application will take effect. Note that replica replication from the source cluster is feasible, and a Failover in the source cluster will not affect the replication of the standby cluster. However, the new cluster cannot create replication slots on the read-only replica, and there may be related error reports and a risk of replication interruption. It is recommended to adjust the upstream replication source of the standby cluster in time.

 standby_cluster:
   create_replica_methods:
   - basebackup
-  host: 10.10.10.13
+  host: 10.10.10.12
   port: 5432

Modify the IP of the replication upstream in standby,_cluster.host, and the application will take effect (no need to reboot, Reload).

Delayed Cluster

HA and M-S replication can solve the problems caused by machine hardware failure, but cannot solve the failure caused by software bugs and human operations. A cold standby is usually required for accidental data deletion, but another way is to prepare a delayed cluster.

You can use the function standby cluster to create a delayed. For example, now you want to specify a delayed for the pg-test cluster: pg-testdelay, which is the state of pg-test 1 hour ago.

# pg-test is the original database
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-test
    pg_version: 14

# pg-testdelay will be used as a delayed for the pg-test
pg-testdelay:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary , pg_upstream: 10.10.10.11 } # The actual role is Standby Leader
  vars:
    pg_cluster: pg-testdelay
    pg_version: 14

After creation, edit the Patroni config file for the delayed cluster using pg edit-config pg-testdelay in the meta node and change standby_cluster.recovery_min_apply_delay to the delay value you expect.

 standby_cluster:
   create_replica_methods:
   - basebackup
   host: 10.10.10.11
   port: 5432
+  recovery_min_apply_delay: 1h

Cascade Instance

When creating a cluster, if the pg_upstream parameter is specified for one of the replicas in the cluster (defined as another replica in the cluster), the instance will attempt to build logical replication from that specified replica.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica } # Try to replicate from slave 2 instead of the master
    10.10.10.13: { pg_seq: 2, pg_role: replica, pg_upstream: 10.10.10.12 }
  vars:
    pg_cluster: pg-test

Citus Deployment

Citus is a distributed extension plugin for PostgreSQL. By default, Pigsty installs Citus but does not enable it. pigsty-citus.yml provides a config file case for deploying a Citus cluster. To allow Citus to, you need to modify the following parameters.

  • max_prepared_transaction: Modify to a value greater than max_connections, e.g. 800.
  • pg_libs: Must contain citus and be placed in the top position.
  • You need to include the citus extension plugin in the business database (but you can also manually install it via CREATE EXTENSION).
Citus cluster sample config 
#----------------------------------#
# cluster: citus coordinator
#----------------------------------#
pg-meta:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary , pg_offline_query: true }
  vars:
    pg_cluster: pg-meta
    vip_address: 10.10.10.2
    pg_users: [ { name: citus , password: citus , pgbouncer: true , roles: [ dbrole_admin ] } ]
    pg_databases: [ { name: meta , owner: citus , extensions: [ { name: citus } ] } ]

#----------------------------------#
# cluster: citus data nodes
#----------------------------------#
pg-node1:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-node1
    vip_address: 10.10.10.3
    pg_users: [ { name: citus , password: citus , pgbouncer: true , roles: [ dbrole_admin ] } ]
    pg_databases: [ { name: meta , owner: citus , extensions: [ { name: citus } ] } ]

pg-node2:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary  , pg_offline_query: true }
  vars:
    pg_cluster: pg-node2
    vip_address: 10.10.10.4
    pg_users: [ { name: citus , password: citus , pgbouncer: true , roles: [ dbrole_admin ] } ]
    pg_databases: [ { name: meta , owner: citus , extensions: [ { name: citus } ] } ]

pg-node3:
  hosts:
    10.10.10.13: { pg_seq: 1, pg_role: primary  , pg_offline_query: true }
  vars:
    pg_cluster: pg-node3
    vip_address: 10.10.10.5
    pg_users: [ { name: citus , password: citus , pgbouncer: true , roles: [ dbrole_admin ] } ]
    pg_databases: [ { name: meta , owner: citus , extensions: [ { name: citus } ] } ]

Next, you need to refer to the Citus Multi-Node Deployment Guide, and on the Coordinator node, execute the following command to add a data node.

sudo su - postgres; psql meta
SELECT * from citus_add_node('10.10.10.11', 5432);
SELECT * from citus_add_node('10.10.10.12', 5432);
SELECT * from citus_add_node('10.10.10.13', 5432);
SELECT * FROM citus_get_active_worker_nodes();
  node_name  | node_port
-------------+-----------
 10.10.10.11 |      5432
 10.10.10.13 |      5432
 10.10.10.12 |      5432
(3 rows)

After successfully adding data nodes, you can use the following command to create sample data tables on the coordinator and distribute them to each data node.

-- Declare a distributed table
CREATE TABLE github_events
(
    event_id     bigint,
    event_type   text,
    event_public boolean,
    repo_id      bigint,
    payload      jsonb,
    repo         jsonb,
    actor        jsonb,
    org          jsonb,
    created_at   timestamp
) PARTITION BY RANGE (created_at);
-- Creating Distributed Tables
SELECT create_distributed_table('github_events', 'repo_id');

For more information about Citus, please refer to the Citus official doc.

MatrixDB Deployment

Greenplum is a distributed data warehouse based on the PostgreSQL ecosystem, and MatrixDB is a branch of Greenplum based on Greenplum 7, using the PostgreSQL 12 kernel. Greenplum 7 has not yet been officially released, so Pigsty is currently using MatrixDB as a replacement for Greenplum.

MatrixDB is based on the PostgreSQL ecosystem, so most PostgreSQL playbooks and tasks can be reused on MatrixDB. There are only two additional parameters specific to MatrixDB.

  • gp_role: Define the identity of the Greenplum cluster, master, or segment.
  • pg_instances: Define the Segment instance, which is used to deploy the Segment monitoring instance.

For details, please refer to MatrixDB Deployment.

MatrixDB cluster sample config--4 nodes 
#----------------------------------#
# cluster: mx-mdw (gp master)
#----------------------------------#
mx-mdw:
  hosts:
    10.10.10.10: { pg_seq: 1, pg_role: primary , nodename: mx-mdw-1 }
  vars:
    gp_role: master          # this cluster is used as greenplum master
    pg_shard: mx             # pgsql sharding name & gpsql deployment name
    pg_cluster: mx-mdw       # this master cluster name is mx-mdw
    pg_databases:
      - { name: matrixmgr , extensions: [ { name: matrixdbts } ] }
      - { name: meta }
    pg_users:
      - { name: meta , password: DBUser.Meta , pgbouncer: true }
      - { name: dbuser_monitor , password: DBUser.Monitor , roles: [ dbrole_readonly ], superuser: true }

    pgbouncer_enabled: true                # enable pgbouncer for greenplum master
    pgbouncer_exporter_enabled: false      # enable pgbouncer_exporter for greenplum master
    pg_exporter_params: 'host=127.0.0.1&sslmode=disable'  # use 127.0.0.1 as local monitor host

#----------------------------------#
# cluster: mx-sdw (gp master)
#----------------------------------#
mx-sdw:
  hosts:
    10.10.10.11:
      nodename: mx-sdw-1        # greenplum segment node
      pg_instances:             # greenplum segment instances
        6000: { pg_cluster: mx-seg1, pg_seq: 1, pg_role: primary , pg_exporter_port: 9633 }
        6001: { pg_cluster: mx-seg2, pg_seq: 2, pg_role: replica , pg_exporter_port: 9634 }
    10.10.10.12:
      nodename: mx-sdw-2
      pg_instances:
        6000: { pg_cluster: mx-seg2, pg_seq: 1, pg_role: primary , pg_exporter_port: 9633  }
        6001: { pg_cluster: mx-seg3, pg_seq: 2, pg_role: replica , pg_exporter_port: 9634  }
    10.10.10.13:
      nodename: mx-sdw-3
      pg_instances:
        6000: { pg_cluster: mx-seg3, pg_seq: 1, pg_role: primary , pg_exporter_port: 9633 }
        6001: { pg_cluster: mx-seg1, pg_seq: 2, pg_role: replica , pg_exporter_port: 9634 }
  vars:
    gp_role: segment               # these are nodes for gp segments
    pg_shard: mx                   # pgsql sharding name & gpsql deployment name
    pg_cluster: mx-sdw             # these segment clusters name is mx-sdw
    pg_preflight_skip: true        # skip preflight check (since pg_seq & pg_role & pg_cluster not exists)
    pg_exporter_config: pg_exporter_basic.yml   # use basic config to avoid segment server crash
    pg_exporter_params: 'options=-c%20gp_role%3Dutility&sslmode=disable'  # use gp_role = utility to connect to segments

19 - Redis Deployment

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty is a PostgreSQL distribution and a general-purpose application runtime. It can manage, deploy, and monitor other applications and databases, such as Redis.

Similar to PostgreSQL, deploying Redis requires the same two steps.

  1. Declare/Define the Redis cluster
  2. Execute Playbook to create the Redis cluster

Define Redis Cluster

ER Model

The Redis entity concept model is almost identical to PostgreSQL and includes the Cluster and Instance. Note that Cluster here does not refer to the clusters in Redis’ native clusters.

The core difference is that Redis typically uses multiple singleton instances, with various Redis instances typically deployed on a single physical/VM to take advantage of multi-core CPUs.

In Pigsty-managed Redis, it is not yet possible to deploy two Redis instances from different clusters on a node, but this does not affect the deployment of multiple independent Redis instances on a node.

Identity Parameters

The identity parameters are the information that must be provided when defining a Redis cluster and include.

Name Attribute Description Example
redis_cluster MUST, cluster level Cluster name redis-test
redis_node MUST, node level Node Number 1,2
redis_instances MUST, node level Instance Definition { 6001 : {} ,6002 : {}}

  • redis_cluster: Identifies the Redis cluster name, configured at the cluster level, as the top-level namespace for cluster resources.

  • redis_node: Identifies the number of the node in the cluster.

  • redis_instances: A JSON object with the Key as the instance port and the Value as a JSON object containing the instance-specific configuration.

Cluster Definition

A condensed definition of three Redis clusters is given below, including.

  • A 1-node, 3-instance Redis Sentinel cluster redis-sentinel.
  • A 2-node, 12-instance Redis Cluster redis-cluster.
  • A 1-node, one primary & two replicas Redis Standalone cluster redis-standalone.

It would help to assign a unique port to the Redis instance on the node.

Redis Sentinel Cluster Example 

#----------------------------------#
# redis sentinel example           #
#----------------------------------#
redis-meta:
  hosts:
    10.10.10.10:
      redis_node: 1
      redis_instances:  { 6001 : {} ,6002 : {} , 6003 : {} }
  vars:
    redis_cluster: redis-meta
    redis_mode: sentinel
    redis_max_memory: 128MB

Redis Native Cluster Example 

#----------------------------------#
# redis native cluster example            #
#----------------------------------#
redis-test:
  hosts:
    10.10.10.11:
      redis_node: 1
      redis_instances: { 6501 : {} ,6502 : {} ,6503 : {} ,6504 : {} ,6505 : {} ,6506 : {} }
    10.10.10.12:
      redis_node: 2
      redis_instances: { 6501 : {} ,6502 : {} ,6503 : {} ,6504 : {} ,6505 : {} ,6506 : {} }
  vars:
    redis_cluster: redis-test           # name of this redis 'cluster'
    redis_mode: cluster                 # standalone,cluster,sentinel
    redis_max_memory: 64MB              # max memory used by each redis instance
    redis_mem_policy: allkeys-lru       # memory eviction policy

Redis Standalone Example 

#----------------------------------#
# redis standalone example         #
#----------------------------------#
redis-common:
  hosts:
    10.10.10.13:
      redis_node: 1
      redis_instances:
        6501: {}
        6502: { replica_of: '10.10.10.13 6501' }
        6503: { replica_of: '10.10.10.13 6501' }
  vars:
    redis_cluster: redis-common         # name of this redis 'cluster'
    redis_mode: standalone              # standalone,cluster,sentinel
    redis_max_memory: 64MB              # max memory used by each redis instance

Create Redis Cluster

Playbook

Create a Redis instance/cluster using the playbook redis.yml.

./redis.yml -l redis-sentinel
./redis.yml -l redis-cluster
./redis.yml -l redis-standalone

Caveat

Although not recommended, it is still possible to deploy a mix of PostgreSQL and Redis to make the most of machine resources.

The redis.yml playbook will deploy the Redis Monitor Exporter on the machine, including redis_exporter and node_exporter (optional).

If the machine’s node_exporter exists during this process, it will be redeployed.

By default, Prometheus will use the “multi-target crawl” mode, using the Redis Exporter on port 9121 on the node to crawl all Redis instances on that node.

Checking Redis Monitor

Pigsty currently provides three Redis monitor dashboards as part of a standalone monitoring application REDIS.

  • Redis Overview: Provide a global overview of Redis in the entire environment.
  • Redis Cluster: Focus on monitoring information for a single Redis business cluster.
  • Redis Instance: Focus on detailed monitoring information for a single Redis instance.

You can use the included Redis-benchmark test.

Other Functions

Pigsty v1.5.1 supports whole-cluster Redis deployment and monitoring, and the tagged redis.yml/redis-remove.yml playbooks also target one instance with -e redis_port=<port>.

20 - MatrixDB Deployment

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty can be used to deploy and monitor MatrixDB (equal to Greenplum 7+ Time-Series database).

Currently, MatrixDB uses PostgreSQL 12 kernel, while native Greenplum uses 9.6 kernels, so MatrixDB is used instead of Greenplum implementation, and Greenplum support will be added later.

E-R Model

MatrixDB is logically divided into two parts, Master and Segments, composed of PostgreSQL instances, divided into four categories: Master/Standby/Primary/Mirror.

  • Master is the port directly accessed by the user to take over queries. There is only one MatrixDB deployment, usually deployed using a standalone node.
  • Standby is a physical replica of the Master instance, which takes over when the Master fails and is usually deployed using a standalone node, which is optional.
  • A MatrixDB deployment typically has multiple Segments, each of which consists of a mandatory primary instance and an optional mirror instance.
  • The segment’s primary is responsible for the actual storage and computation, and the mirror does not carry the read and write traffic. It takes over for the primary when the primary is down, usually distributed on different nodes from the primary.
  • MatrixDB installation wizard determines the distribution of primary and mirror of Segment, and there may be different Segment instances on Segments nodes of the cluster.

Deployment conventions

  • Master cluster (master/standby) (gp_role = master) constitutes a PostgreSQL cluster, usually named to contain mdw, e.g., mx-mdw.
  • Each Segment (primary/mirror) (gp_role = segment) constitutes a PostgreSQL cluster, usually named with seg, e.g., mx-seg1, mx-seg2.
  • The user should explicitly name the cluster nodes, e.g., mx-sdw-1, mx-sdw-2, …

Download

The RPM pkgs for MatrixDB & Greenplum are not part of the standard Pigsty deployment and will not be placed in the default pkg.tgz.

The RPM pkgs for MatrixDB & Greenplum and their complete dependencies will be packaged as a separate offline pkg matrix.tgz.

You can add new matrix sources to the Pigsty meta node.

# Download Address（Github）：https://github.com/Vonng/pigsty/releases/download/v1.5.1/matrix.tgz
# Download Address（China CDN）：http://download.pigsty.cc/v1.5.1/matrix.tgz
# Download the script on the meta node, under the pigsty dir, directly using the download matrix to download and unzip
./download matrix

This command creates a /www/matrix.repo file, which by default you can access at http://pigsty/matrix.repo to get the repo, which points to the http://pigsty/matrix.

Configure

The MatrixDB / Greenplum installation will reuse the PGSQL tasks and config with the exclusive config parameters gp_role and pg_instances.

The config file pigsty-mxdb.yml gives a sample deployment of MatrixDB in a four-node sandbox.

Using `configure -m mxdb` will automatically use this config file as a template.
./configure -m mxdb

This config file node_repo_local_urls adds the new Yum repo, and http://pigsty/matrix.repo ensures that all nodes access Matrix Repo.

Execute

Deploy MatrixDB in a four-node sandbox. Note. Otherwise, the default will be to use DBSU mxadmin:mxadmin as the monitoring username and password.

#  If you deploy MatrixDB Master on a meta node, add the no_cmdb option; otherwise, install it normally.
./infra.yml -e no_cmdb=true

# Configure all nodes for MatrixDB installation
./nodes.yml

# Install MatrixDB on the above node
./pgsql-matrixdb.yml

Once the installation is complete, you need to complete the next installation through the WEB UI provided by MatrixDB. Open http://mx.pigsty or visit http://10.10.10.10:8240 and fill in pgsql-matrixdb.yml with the initial user password output at the end to enter the installation wizard.

Follow the prompts to add the MatrixDB nodes: 10.10.10.11, 10.10.10.12, 10.10.10.13, click Confirm Installation and proceed to the next step.

Monitoring uses mxadmin:mxadmin as the monitoring username password by default. Please fill in mxadmin or your password.

If a different password was specified in the installation wizard, change the pg_monitor_username and pg_monitor_password variables (using another user than dbsu, additional HBAs will usually need to be configured on all instances as well).

Note that the logic for MatrixDB / Greenplum to assign Segments on nodes is currently uncertain. Once initialization is complete, you can modify the definition of Segment instances in pg_instances and redeploy monitoring to reflect the true topology.

Post-Run

Finally, manually execute the following command on the Greenplum/MatrixDB Master node to allow the monitoring component to access the replica and restart it to take effect.

sudo su - mxadmin
psql postgres -c "ALTER SYSTEM SET hot_standby = on;"  # Configure hot_standby=on to allow queries from the replica
gpconfig -c hot_standby -v on -m on                    # Configure hot_standby=on to allow queries from the replica
gpstop -a -r -M immediate                              # Restart MatrixDB immediately to take effect

All MatrixDB clusters can then be observed from the monitoring system. The MatrixDB Dashboard provides an overview of the overall monitoring of the data warehouse.

Optional

You can treat MatrixDB’s Master cluster as a standard PostgreSQL cluster and use pgsql-createdb with pgsql-createuser to create a business database with users.

bin/createuser mx-mdw  dbuser_monitor   # Create a monitoring user on Master
bin/createdb   mx-mdw  matrixmgr        # Create a dedicated database for monitoring on the Master
bin/createdb   mx-mdw  meta             # Create a new database on the Master

21 - Playbooks

Historical documentation restored from the Pigsty v1.5.1 tag.

Learn about the pre-built playbooks provided by Pigsty, the features, how to use them, and the considerations.

Pigsty implements core control functions at the bottom through the Ansible Playbook, and Pigsty provides pre-built playbooks in four main categories:

  • infra: Use the infra series of playbooks to install Pigsty singleton on the meta node with optional features.
  • nodes: Use the nodes series of playbooks to include more nodes in Pigsty monitoring and management and for subsequent use.
  • pgsql: Use the pgsql series of playbooks to deploy and manage PostgreSQL database clusters on existing nodes.
  • redis: Use the redis series of playbooks to deploy and manage various modes of Redis clusters on existing nodes.

Overview

Playbook Function Link
infra Full installation of Pigsty on the meta node src
infra-demo Special playbook for complete initialization of a four-node demo sandbox in one go src
infra-jupyter Adding the optional data analysis service component Jupyter Lab to the meta node src
nodes Node provisioning to include nodes in Pigsty for subsequent database deployment src
nodes-remove Node remove, unloading node DCS and monitoring, no longer included in Pigsty src
pgsql PostgreSQL cluster deploy, or expand src
pgsql-remove PostgreSQL cluster destruction, or downsize src
pgsql-createuser Creating PostgreSQL business users src
pgsql-createdb Creating a PostgreSQL Business Database src
pgsql-monly Monly mode, with access to existing PostgreSQL instances or RDS src
pgsql-migration Generate PostgreSQL semi-automatic database migration solution (Beta) src
pgsql-matrixdb Reuse the PG role to deploy a MatrixDB data warehouse clusters (Beta) src
redis Deploy a Redis database in cluster/standalone/Sentinel mode src
redis-remove Redis cluster/node destruction src

The typical use process is as follows：

  1. Use the infra series of playbooks to install Pigsty on the meta node/local machine and deploy the infra.

    All playbooks initiate execution on the meta node, and the infra series of playbooks only works on the meta node.

  2. Use the nodes series of playbooks to include or remove other nodes from Pigsty.

    After a node is managed, node monitoring and logging can be accessed from the meta node Grafana, and the node joins the Consul cluster.

  3. Use the pgsql series of playbooks to deploy a PostgreSQL cluster on managed nodes.

    After deployment on the managed node, you can access PostgreSQL monitoring and logs from the meta node.

  4. Use the redis series of playbooks to deploy a Redis cluster on managed nodes.

    After deployment on the managed node, Redis monitoring and logs can be accessed from the meta node.

                                           meta     node
[infra.yml]  ./infra.yml [-l meta]        +pigsty
[nodes.yml]  ./nodes.yml -l pg-test                 +consul +monitor
[pgsql.yml]  ./pgsql.yml -l pg-test                 +pgsql
[redis.yml]  ./redis.yml -l pg-test                 +redis

Most playbooks are idempotent, meaning that some deployment playbooks may erase existing databases and create new ones without the protection option turned on.

Please read the documentation carefully, proofread the commands several times, and operate with caution. The author is not responsible for any loss of databases due to misuse.

Ansible Quick Start

The Pigsty playbooks are written in Ansible.

  • Ansible Installation: How to install Ansible? (Pigsty users usually don’t have to worry about）
  • Limit Host: How to execute a playbook for a limit host?
  • Task Subset: How to perform certain specific tasks in the playbook？
  • Extra Params: How to pass in extra command-line params to control playbook behavior？

Installation

The Ansible playbook requires the ansible-playbook executable command, and Ansible can be installed on EL7-compatible systems with the following command.

yum install ansible

Pigsty will attempt to install ansible from the offline package when using offline packages during the Configure phase.

There are three core params to focus on when executing the playbook：-l|-t|-e, which are used to restrict the host for execution, with the task to be performed and to pass in extra params, respectively.

Limit Host

The target of execution can be selected with the -l|-limit <selector> param. When this param is not specified, most playbooks default to all hosts defined in the configuration file as the target of execution. It is highly recommended to specify the execution object when executing the playbook.

There are two types of objects commonly used, clusters and hosts.

./pgsql.yml                 # Execute the pgsql playbook on all inventory hosts(this is dangerous!)
./pgsql.yml -l pg-test      # Execute the pgsql playbook against the hosts in the pg-test cluster
./pgsql.yml -l 10.10.10.10  # Execute the pgsql playbook against the host at 10.10.10.10
./pgsql.yml -l pg-*         # Execute the playbook against a cluster that matches the pg-* pattern (glob)

Task Subset

You can select the task subset to be executed with -t|--tags <tags>. When this param is not specified, the full playbook will be executed, and the selected task subset will be executed when set.

./pgsql.yml -t pg_hba                            # Regenerate and apply cluster HBA rules

Users can separate each task by , and perform multiple tasks at once. For example, you can adjust the cluster LB configuration using the following command when the cluster role members change.

./pgsql.yml -t haproxy_config,haproxy_reload     # Regenerate the cluster LB configuration and apply

Extra Params

Extra command-line params can be passed in via -e|-extra-vars KEY=VALUE to override existing params or control some special behavior.

For example, some of the behavior of the following playbooks can be controlled via command-line params.

./nodes.yml -e ansible_user=admin -k -K      # When configuring the node, use another admin user, and enter ssh with the sudo password
./pgsql.yml -e pg_clean=true        # Force erase existing running database instances when installing PG (dangerous)
./infra-remove.yml -e rm_metadata=true       # Remove data when uninstalling Pigsty
./infra-remove.yml -e rm_metapkgs=true      # Uninstall the software when uninstalling Pigsty
./nodes-remove.yml -e dcs_safeguard=false     # When removing a node, force removal even if there is a DCS server on it
./pgsql-remove.yml -e rm_pgdata=true         # When removing PG, remove data together
./pgsql-remove.yml -e rm_pgpkgs=true         # When removing the PG, uninstall the software as well

22 - Playbook: INFRA

Historical documentation restored from the Pigsty v1.5.1 tag.

Install pigsty & Add new optional to meta node with INFRA series playbooks.

Playbook Function Link
infra Install pigsty on meta nodes src
infra-demo Special version of infra, pull up the 4-node sandbox in one-pass src
infra-remove Uninstall pigsty from the meta node. src
infra-jupyter Install optional JupyterLab on meta node src

infra

The infra.yml playbook will install Pigsty on the meta nodes.

When using Pigsty as a battery-included Postgres, infra.yml is sufficient for the installation.

What

The following tasks will be executed:

  • Configure dir and environment variables of meta nodes.
  • Download and create a LocalYum repo. (If offline packages /tmp/pkg.tgz exists, skip the download phase)
  • Init the current meta node as a common node in Pigsty management.
  • Deploy infra, including Prometheus, Grafana, Loki, Alertmanager, Consul Server, etc…
  • Init a common PostgreSQL singleton on the current node as CMDB.

Where

This playbook works on meta nodes.

  • Pigsty will use the node currently executing this playbook as Pigsty’s meta node by default.
  • Pigsty will mark the current node as the meta node by default during configure and replace the placeholder IP 10.10.10.10 in the config template with the current node’s primary IP.
  • Meta node can initiate management and deploy infra. It is not different from a regularly managed pgsql node.
  • Pigsty uses meta nodes by default to deploy DCS Servers for PostgreSQL HA. But you can use external DCS Servers, too.
  • Using multiple meta nodes is possible. Refer to the pigsty-dcs3 template: 3 meta x 3 nodes, allowing one failure.

How

Some special notes:

  • This playbook is idempotent. Play again will erase Consul Server and CMDB (when protection is disabled)
  • The complete execution of this playbook takes about 5-8 minutes with offline pkgs, depending on your machine spec.
  • Depending on your network conditions, downloading packages directly from the Internet may take 10-20 minutes.
  • This playbook treats the meta node as a common node to manage and deploy the PGSQL database. If it can be executed successfully, you could be confident that nodes.yml and pgsql.yml will work on nodes with the same state.
  • The default pg-meta on the meta node will be used as the Pigsty CMDB and supporting advanced features.

Tasks

The playbook.

./infra.yml --tags=environ                       # Reconfigure envs on the meta node
./infra.yml --tags=repo -e repo_rebuild=true     # Forced re-creation of local repo
./infra.yml --tags=repo_upstream                 # Add upstream yum repos
./infra.yml --tags=prometheus                    # Recreate Prometheus
./infra.yml --tags=nginx_config,nginx_restart    # Regenerate Nginx conf and restart
……

Nodes within the meta group will be marked with the meta_node flag in Pigsty inventory.

infra-demo

The infra-demo.yml is a special playbook for the demo env, which can be used to initialize a 4-node sandbox in one go by interweaving meta node & other nodes initialization. It’s useful when trying to deploy multiple meta nodes.

It’s equivalent to following commands in the standard 4-node demo.

./infra.yml              # Install Pigsty on meta node
./nodes.yml -l pg-test   # Init 3 nodes of cluster pg-test
./pgsql.yml -l pg-test   # Init pgsql database cluster pg-test
Warning

USE THIS WITH CAUTION! You could just destroy the entire env with a fat finger & wrong config.

infra-remove

The infra-remove.yml playbook will uninstall pigsty from the meta node.

Following services will be stopped & removed from meta nodes.

  • grafana-server
  • prometheus
  • alertmanager
  • node_exporter
  • consul
  • loki
  • promtail

infra-jupyter

Playbook infra-jupyter.yml will install JupyterLab on the meta node.

Check Tutorial: Jupyter Lab for more details.

23 - Playbook: NODES

Historical documentation restored from the Pigsty v1.5.1 tag.

Use the NODES playbook to bring more nodes to Pigsty, adjusting nodes to the state described in the config.

Once pigsty is installed on the meta node with infra.yml, You can add more nodes to Pigsty with nodes.yml or remove them from Pigsty with nodes-remove.yml.

Playbook Function Link
nodes Node Provisioning. Register node into Pigsty and prepare for database deployment src
nodes-remove Node Removal, uninstall DCS & Monitoring & Logging, de-register from Pigsty src

nodes

The nodes.yml playbook will register nodes to Pigsty.

This playbook adjusts the target nodes to the state described in the inventory, installs the Consul service, and incorporates it into the Pigsty monitoring system. Nodes can be used for database deployment once provisioning is complete.

The behavior of this playbook is determined by the Config: NODES. The complete execution of this playbook may take 1 to 3 minutes when using the local yum repo, depending on the machine spec.

./nodes.yml             # init all nodes in inventory    (danger!)
./nodes.yml -l pg-test  # init nodes under group pg-test (recommended!)
./nodes.yml -l pg-meta,pg-test   # init nodes in both clusters: pg-meta and pg-test
./nodes.yml -l 10.10.10.11       # init node with ip address 10.10.10.11

This playbook will run the following tasks:

  • Generate node identity parameters
  • Provisioning Node
    • Configure the node’s hostname
    • Configure static DNS records
    • Configure dynamic DNS resolver
    • Configure yum repo
    • Install specified RPM packages
    • Configure features such as NUMA/SWAP/firewall
    • Configure node tuned tuning templates
    • Configure shortcuts and environment variables for the node
    • Create node admin user and configure its SSH access
    • Configure timezone
    • Configure NTP service
  • Initialize the DCS service on the node: Consul
    • Erase existing Consul if it exists (with protection disabled)
    • Initialize the Consul Agent or Server service for the current node
  • Initialize the node monitoring component and incorporate Pigsty
    • Install Node Exporter
    • Register Node Exporter to Prometheus on meta nodes.
Warning

Be careful when running this playbook on provisioned nodes. It may lead to the database being temporarily unavailable because of the removal of the consul service.

The dcs_clean provides a SafeGuard to avoid accidental purge. When existing Consul Instance is detected during playbook execution. It will take action about it.

Warning

When using the complete nodes.yml playbook or just the section on dcs|consul, please double-check that the -tags|-t and -limit|-l is correct. Make sure you are running the right tasks on the correct targets.

SafeGuard

Pigsty provides a SafeGuard to avoid purging running consul instances with fat fingers. There are two parameters.

  • dcs_safeguard: Disabled by default, if enabled, running consul will not be purged by any circumstance.
  • dcs_clean: the Consul role fallback and bundled v1.5.1 sandbox inventory set this to true; set it to false and enable the safeguard in protected environments.

When running consul exists, nodes.yml will act as:

dcs_safeguard / dcs_clean dcs_clean=true dcs_clean=false
dcs_safeguard=true ABORT ABORT
dcs_safeguard=false PURGE ABORT

When running consul exists, nodes-remove.yml will act as:

dcs_safeguard / dcs_clean dcs_clean=true dcs_clean=false
dcs_safeguard=true ABORT ABORT
dcs_safeguard=false PURGE PURGE

Selective Execution

You can selectively execute a subset of this playbook through tags.

For example, if you want to re-deploy node monitor components only:

./nodes.yml --tags=node-monitor

Common tasks are listed below:

# play
./nodes.yml --tags=node-id         # generate & print node identity params
./nodes.yml --tags=node-init       # provisoning the node
./nodes.yml --tags=node-dcs        # init dcs on node
./nodes.yml --tags=node-monitor    # init monitor (metrics & logs) on node

# tasks
./nodes.yml --tags=node_name       # Configure node‘s hostname
./nodes.yml --tags=node_dns        # Configure node's static DNS records
./nodes.yml --tags=node_resolv     # Configuring Dynamic DNS Resolver
./nodes.yml --tags=node_repo       # Configure yum repo
./nodes.yml --tags=node_pkgs       # Install specified RPM package
./nodes.yml --tags=node_feature    # Configure NUMA/SWAP/FIREWALL...
./nodes.yml --tags=node_tuned      # Configure tuned tuning templates
./nodes.yml --tags=node_profile    # Configure shortcuts & env variables
./nodes.yml --tags=node_admin      # Create node admin user and configure SSH access
./nodes.yml --tags=node_timezone   # Configure node time zone
./nodes.yml --tags=node_ntp        # Configure NTP service
./nodes.yml --tags=docker          # Configure dockerd daemon
./nodes.yml --tags=consul          # Configure consul agent/server
./nodes.yml --tags=consul -e dcs_clean=true   # Force consul reinit

./nodes.yml --tags=node_exporter   # Configure node_exporter on the node and register it
./nodes.yml --tags=node_register   # Registering node monitoring to a meta node
./nodes-remove.yml --tags=register # Deregister node monitoring from meta node

Admin User Provision

Admin user provisioning is a chicken-and-egg problem. To execute playbooks, you need to have an admin user. To create a dedicated admin user, you need to run this playbook.

Pigsty recommends leaving admin user provisioning to your vendor. It’s common to deliver the node with an admin user with ssh & sudo access.

It may require a password to execute ssh & sudo. You can pass them via extra params --ask-pass|-k and --ask-become-pass|-K, entering SSH and sudo password when prompted. You can create a dedicated admin user (with no pass sudo & ssh) with another admin user (with password sudo & ssh).

The following parameters are used to describe the dedicated admin user.

./nodes.yml -t node_admin -l <target_hosts> --ask-pass --ask-become-pass

The default admin user is dba (uid=88). Please do not use postgres or {{ dbsu }} as the admin user. Please try to avoid using root as the admin user directly.

The default user vagrant in the local sandbox has been provisioned with nopass ssh & sudo. You can use vagrant to ssh to all other nodes from the sandbox meta node.

./nodes.yml --limit <target_hosts>  --tags node_admin  \
            -e ansible_user=<another_admin> --ask-pass --ask-become-pass

Refer to: Prepare: Admin User for more details.

nodes-remove

The nodes-remove.yml playbook is used to remove nodes from Pigsty.

The playbook needs to be executed on meta nodes, and targeting nodes need to be removed.

./nodes-remove.yml               # Remove all nodes (dangerous!)
./nodes-remove.yml -l nodes-test # Remove nodes under group nodes-test
./nodes-remove.yml -l 10.10.10.11 # Remove node 10.10.10.11
./nodes-remove.yml -l 10.10.10.10 -e dcs_safeguard=false # Remove even if there is a DCS server

Task Subsets 

# play
./nodes-remove.yml --tags=register      # Remove node registration
./nodes-remove.yml --tags=node-exporter # Remove node metrics collector
./nodes-remove.yml --tags=promtail      # Remove Promtail log agent
./nodes-remove.yml --tags=consul        # Remove Consul Agent service
./nodes-remove.yml --tags=docker        # Remove Docker service
./nodes-remove.yml --tags=consul -e dcs_safeguard=false # Remove Consul (Including Server!)

24 - Playbook: PGSQL

Historical documentation restored from the Pigsty v1.5.1 tag.

Pull up a defined cluster of HA PostgreSQL cluster using the PGSQL series playbook.

Overview

Playbook Function Link
pgsql Deploy a PostgreSQL cluster, or cluster expand src
pgsql-remove Destroy PostgreSQL cluster, or cluster downsize src
pgsql-createuser Create PostgreSQL business users src
pgsql-createdb Create a PostgreSQL Business Database src
pgsql-monly Monly mode, with access to existing PostgreSQL instances or RDS src
pgsql-migration Generate PostgreSQL semi-automatic database migration solution (Beta) src
pgsql-matrixdb Reuse the PG to deploy a set of MatrixDB clusters (Beta) src

pgsql

After completing the infra initialization, users can use pgsql.yml to complete the initialization of the database cluster.

Complete the cluster definition in the Pigsty configuration file and then apply the changes to the environment by executing pgsql.yml.

./pgsql.yml                      # Perform cluster initialization on all machines in the list (Danger!)
./pgsql.yml -l pg-test           # Perform cluster initialization on the machines under the pg-test group (recommended!)
./pgsql.yml -l pg-meta,pg-test   # Initialize both pg-meta and pg-test clusters
./pgsql.yml -l 10.10.10.11       # Initialize the instance on the machine 10.10.10.11

This playbook accomplishes the following.

  • Install, deploy, and initialize PostgreSQL, Pgbouncer, Patroni (postgres).
  • Install the PostgreSQL monitor (monitor).
  • Install and deploy Haproxy and VIP, expose services (service).
  • Register the database instance to the infra to be monitored (register).

This playbook can be misused to accidentally delete the database, as initializing the database will erase the existing database.

The insurance param prevents accidental deletion by allowing automatic aborting or skipping of high-risk operations during initialization when an existing running instance is detected.

Nevertheless, when using pgsql.yml, double-check that -tags|-t and -limit|-l is correct.

Cautions

  • It is strongly recommended to add the -l parameter to the execution to limit the scope of the command execution.
  • When performing initialization for a replica, the user must ensure that the primary has completed initialization.
  • If Patroni takes too long to pull up a replica when a cluster is expanded, the Ansible playbook may abort due to a timeout. (However, making the replica will continue, for example, in scenarios where making the replica takes more than one day).
  • It is possible to perform subsequent steps from the -Wait for patroni replica online task via Ansible’s -start-at-task after the replica has been automatically crafted. Please refer to SOP for details.

SafeGuard

Pigsty provides a SafeGuard to avoid purging running PostgreSQL instances with fat fingers. There are two parameters.

  • pg_safeguard: Disabled by default, if enabled, running PostgreSQL will not be purged by any circumstance.
  • pg_clean: the role fallback is false, while the bundled disposable v1.5.1 sandbox inventory sets it to true; protected environments should disable it and enable the safeguard.

When running pg exists, pgsql.yml will act as:

pg_safeguard / pg_clean pg_clean=true pg_clean=false
pg_safeguard=true ABORT ABORT
pg_safeguard=false PURGE ABORT

When running pg exists, pgsql-remove.yml will act as:

pg_safeguard / pg_clean pg_clean=true pg_clean=false
pg_safeguard=true ABORT ABORT
pg_safeguard=false PURGE PURGE

Selective execution

An ansible’s tagging mechanism can select a subset of the execution playbook.

For example, if you want to perform only service initialization, you can use the following command.

./pgsql.yml --tags=service      # Refreshing the service definition of a cluster

The common subsets of commands are as follows.

# Infra initialization
./pgsql.yml --tags=infra        # Complete infra initialization, including machine node initialization and DCS deployment


# Database initialization
./pgsql.yml --tags=pgsql        # Complete database deployment: database, monitoring, services

./pgsql.yml --tags=postgres     # Complete database deployment
./pgsql.yml --tags=monitor      # Complete monitoring deployment
./pgsql.yml --tags=service      # Complete load balancing deployment（Haproxy & VIP）
./pgsql.yml --tags=register     # Registering services to the infra

Daily management tasks

Daily management can also be used ./pgsql.yml to modify the state of the cluster. The common command subsets are as follows.

./pgsql.yml --tags=node_admin           # Create an admin user on the target node

# If the current administrator does not have ssh to the target node, you can use another user with ssh to create an administrator (enter the password)
./pgsql.yml --tags=node_admin -e ansible_user=other_admin -k

./pgsql.yml --tags=pg_scripts           # Update the /pg/bin/ directory script
./pgsql.yml --tags=pg_hba               # Regenerate and apply cluster HBA rules
./pgsql.yml --tags=pgbouncer            # Reset Pgbouncer
./pgsql.yml --tags=pg_user              # Full volume refresh business users
./pgsql.yml --tags=pg_db                # Full volume refresh of business database

./pgsql.yml --tags=register_consul      # Register the Consul service locally with the target instance (local execution)
./pgsql.yml --tags=register_prometheus  # Register monitoring objects in Prometheus (proxy to all Meta nodes for execution)
./pgsql.yml --tags=register_grafana     # Register monitoring objects in Grafana (only once)
./pgsql.yml --tags=register_nginx       # Register a LB with Nginx (proxy to all Meta nodes for execution)

# Redeploy monitoring using binary installation
./pgsql.yml --tags=monitor -e exporter_install=binary

# Refresh the service definition of the cluster (changes in cluster membership or service definition)
./pgsql.yml --tags=haproxy_config,haproxy_reload

pgsql-remove

Database Destruction: Remove existing database cluster or instance, reclaim node: pgsql-remove.yml.

The pgsql-remove.yml is the reverse of pgsql.yml and will do the following ：

  • Unregister the database instance from the infra（register）
  • Stop the LB, service component（service）
  • Removal of monitoring system components（monitor）
  • Remove Pgbouncer, Patroni, Postgres（postgres）
  • Remove database dir（rm_pgdata: true）
  • Remove packages (rm_pgpkgs: true)

The playbook has two command-line options to remove the database dir and packages (the default destruction does not remove data and packages).

rm_pgdata: false        # remove postgres data? false by default
rm_pgpkgs: false        # uninstall pg_packages? false by default

Daily management 

./pgsql-remove.yml -l pg-test          # Destruction pg-test cluster
./pgsql-remove.yml -l 10.10.10.13      # Destruction instance 10.10.10.13 (pg-test.pg-test-3)
./pgsql-remove.yml -l 10.10.10.13 -e rm_pgdata=true # Destruction and remove the data dir (slow)
./pgsql-remove.yml -l 10.10.10.13 -e rm_pgpkgs=true   # Destruction and remove the installed PG-related packages

pgsql-createdb

Created business database: Create a new database in an existing cluster or modify a current database: pgsql-createdb.yml.

To ensure that, the author recommends creating a new database in an existing cluster via a playbook or scripting tool.

  • The inventory is consistent with the actual situation.
  • Pgbouncer connection pools are consistent with the database.
  • The data sources registered in Grafana are consistent with the actual situation.

Daily management

Please refer to the section Database for the creation of the database.

# Create a database named test in the pg-test cluster
./pgsql-createdb.yml -l pg-test -e pg_database=test

Simplify commands using wrapper scripts:

bin/createdb <pg_cluster> <dbname>

pgsql-createuser

Create business users: Create a new user or modify an existing user in an existing cluster：pgsql-createuser.yml.

Daily management

Please refer to the section User for the creation of business users.

# Create a user named test in the pg-test cluster
./pgsql-createuser.yml -l pg-test -e pg_user=test

Simplify commands using wrapper scripts.

bin/createuser <pg_cluster> <username>

Note that the user-specified by pg_user must already be in the definition of the cluster pg_users. Otherwise, an error will be reported.

pgsql-monly

Dedicated playbook for performing monitoring deployments. See monly deployments for details.

pgsql-matrixdb

Dedicated playbook for deploying MatrixDB. See Deploying MatrixDB Cluster for details.

pgsql-migration

Playbook for automated database migration, still in Beta status. See database cluster migration for details.

25 - Playbook: REDIS

Historical documentation restored from the Pigsty v1.5.1 tag.

REDIS series Playbook: Define and pull Redis databases in traditional standalone, native, and sentinel clusters.

Playbook Function Link
redis Deploying a Redis database in Native/Standalone/Sentinel cluster src
redis-remove Redis cluster/node destruction src

redis

Deploy redis instances on nodes.

# init all redis instances on group <cluster>
 ./redis.yml -l <cluster>       # create redis among group <cluster>

# init all redis instances specific node
 ./redis.yml -l 10.10.10.10     # setup all redis instances on node 10.10.10.10

# init one specific instance 10.10.10.11:6501 (skip redis node)
 ./redis.yml -l 10.10.10.11 -e redis_port=6501 -t redis

Alias script bin/createredis wrap above playbook with:

bin/createredis redis-common            # init redis cluster redis-common
bin/createredis 10.10.10.10             # init redis node 10.10.10.10
bin/createredis 10.10.10.13 6501 6502   # init redis instance 10.10.10:13:6501 10.10.10:13:6502

redis-remove

Remove redis instances from nodes.

# Remove cluster `redis-test`
redis-remove.yml -l redis-test

# Remove all instance on redis node 10.10.10.13
redis-remove.yml -l 10.10.10.13

# Remove one specific instance 10.10.10.13:6501
redis-remove.yml -l 10.10.10.13 -e redis_port=6501

26 - Configuration

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty uses declarative configuration.

Pigsty defines infra and database clusters through Inventory, and each Pigsty deploy has a corresponding config. Pigsty’s config uses the “Infra as Data” philosophy: Users describe requirements through the declarative config, and Pigsty adapts the fundamental components to the expected state.

Formally, the inventory can be implemented as a default local config file or as dynamic configuration data in CMDB. This article uses the default YAML configuration file pigsty.yml as an example. Pigsty detects the current node environment and generates the recommended config file in configure.

The main content of the inventory is config entries. The source-backed v1.5.1 summary below contains 213 parameters that can be configured at multiple levels, and most parameters can use default values. Config entry can be divided into four major categories according to category: INFRA， NODES/host nodes， PGSQL， REDIS, and further subdivided into 32 subcategories.

Configure

Go to the Pigsty project dir and execute configure. Pigsty will generate a config file based on the current machine env, a process called Configure.

./configure [-n|--non-interactive] [-d|--download] [-i|--ip <ipaddr>] [-m|--mode {auto|demo}]

configure will check the following things, minor problems will be fixed automatically, otherwise, it will prompt an error to exit.

check_kernel     # kernel        = Linux
check_machine    # machine       = x86_64
check_release    # release       = CentOS 7.x
check_sudo       # current_user  = NOPASSWD sudo
check_ssh        # current_user  = NOPASSWD ssh
check_ipaddr     # primary_ip (arg|probe|input)              (INTERACTIVE: ask for ip)
check_admin      # check current_user@primary_ip nopass ssh sudo
check_mode       # check machine spec to determine node mode (tiny|oltp|olap|crit)
check_config     # generate config according to primary_ip and mode
check_pkg        # check offline installation package exists (INTERACTIVE: ask for download)
check_repo       # create repo from pkg.tgz if exists
check_repo_file  # create local file repo file if repo exists
check_utils      # check ansible sshpass and other utils installed

Running ./configure directly will launch an interactive CLI wizard that prompts the user to answer the following 3 questions:

IP address

When multiple NICs with multiple IPs are detected on the current machine, the config wizard prompts you to enter the primary IP used, which is the IP you use to access the node from the internal network. Note that you should not use the public IP.

Download Package

When offline package /tmp/pkg.tgz not exists on the node, the config wizard will ask whether to download it from Github. Selecting Y will start the download, and selecting N will skip it. If your node has good Internet access with a suitable proxy config, or if you need to make offline packages, you can choose N.

Config Template

The config wizard automatically selects a config template based on the current machine env. However, you can specify the use of a config template manually with -m <mode>.

  • demo: The project’s default config file, the one used by the 4-node sandbox, enables all features.
  • auto: Suitable for deployment in production env with more stable and conservative configs.
  • In addition, Pigsty has several preconfigured config templates that can be specified and used directly with the -m, see the files/conf for details.

The most important part of the config template is to replace the placeholder IP 10.10.10.10 in the template with the real IP (intranet primary IP) of the current machine and select the appropriate database specification template according to the current machine config. You can use the default generated config file directly or make further customization based on the automatically generated config file.

Standard output of the configure 
$ ./configure
configure pigsty v1.5.1 begin
[ OK ] kernel = Linux
[ OK ] machine = x86_64
[ OK ] release = 7.8.2003 , perfect
[ OK ] sudo = root ok
[ OK ] ssh = [email protected] ok
[ OK ] primary_ip = 10.10.10.10  (from probe)
[ OK ] admin = [email protected] ok
[ OK ] spec = mini (cpu = 2)
[ OK ] config = auto @ 10.10.10.10
[ OK ] cache = /tmp/pkg.tgz exists
[ OK ] repo = /www/pigsty ok
[ OK ] repo file = /etc/yum.repos.d/pigsty-local.repo
[ OK ] utils = install from local file repo
[ OK ] ansible = ansible 2.9.27
configure pigsty done. Use 'make install' to proceed

Config File

A specific sample config file is available at the root of the Pigsty project: pigsty.yml.

The top level of the config file is a single object with a key as all and contains two sub-projects: vars and children.

all:                      # Top-level object: all
  vars: <123 keys>        # Global Config: all.vars

  children:               # Grouping Definition: all.children Each project defines a cluster
    meta: <2 keys>...     # Special grouping: meta  Defined environment meta nodes

    pg-meta: <2 keys>...  # Detailed definition of database cluster pg-meta
    pg-test: <2 keys>...  # Detailed definition of database cluster pg-test
    ...

The content of vars is a K-V pair that defines the global config parameters, K is the name of the config entry and V is the content.

The content of children is also a K-V pair, K is the cluster name and V is the specific cluster definition, a sample cluster definition is shown below:

  • The cluster definition also includes two sub-projects: vars defines the config at the cluster level. hosts define the cluster’s instance members.
  • The params in the cluster config override the global params, and the cluster configuration params are overridden by the configuration params of the same name at the instance level. The only mandatory cluster configuration parameter is pg_cluster, which is the name of the cluster and is consistent with the upper-level cluster name.
  • The hosts use K-V to define the cluster instance members, K is the IP (must be ssh reachable), and V is the specific instance config params.
  • There are two mandatory params in the instance config: pg_seq, and pg_role, which are the unique serial number of the instance and the role of the instance, respectively.
pg-test:                 # The cluster name is used as the cluster name by default
  vars:                  # Database cluster level variables
    pg_cluster: pg-test  # A mandatory config entry defined at the cluster level, consistent throughout pg-test.
  hosts:                 # Database Cluster Members
    10.10.10.11: {pg_seq: 1, pg_role: primary} # Database Instance Members
    10.10.10.12: {pg_seq: 2, pg_role: replica} # The identity parameters pg_role and pg_seq must be defined
    10.10.10.13: {pg_seq: 3, pg_role: offline} # Variables at the instance level can be specified here

Pigsty config files follow Ansible rules in YAML format and use a single config file by default. The default config file path is pigsty.yml in the root dir of the Pigsty source code. The default config file is specified via inventory = pigsty.yml in ansible.cfg in the same dir. Additional config files can be specified via -i <config_path> when executing any playbook.

The config file needs to be used in conjunction with Ansible. Ansible is a popular DevOps tool. If you are proficient in Ansible, you can adapt the config file organization and structure according to Ansible’s manifest organization rules.

Please use the browse Ansible Quick Start and start executing playooks with Ansible.

Config Entry

Config entries are in the form of K-V pairs: the key is the name of the Config entry and the value is the content of the config entry.

Pigsty’s params can be configured at different levels and inherited and overwritten based on rules, with higher priority config entries overwriting lower priority config entries with the same name.

Config Entry Levels

In Pigsty’s config file, config entry can appear in three locations: global, cluster, and instance. Config entry defined in cluster vars override global config entry with same-name key override, and config entry defined in an instance, in turn, override cluster config entry with global config entry.

Granularity Scope Priority Description Location
Global Global Low Consistent within the same set of deployment envs all.vars.xxx
Cluster Cluster Medium Consistency within the same set of clusters all.children.<cls>.vars.xxx
Instance Instance High The most granular level of config all.children.<cls>.hosts.<ins>.xxx

Not all config entries are suitable for use at all levels. For example, infra params will usually only be defined in the global config, params such as database instance labels, roles, load balancing weights, and other params can only be configured at the instance level, and some operational options can only be provided using CLI params. For details of config entry, please see the list of config entry.

Default & Overwrite

In addition to the three config granularities, there are two extra levels of priority in the Pigsty config entry: default value and CLI param forced override:

  • Default: When a config entry does not appear at either the global/cluster/instance level, the default config entry is used. The default value has the lowest priority. The default params are defined in roles/<role>/defaults/main.yml.
  • Parameter: Config entry specified by means of CLI incoming params have the highest priority and will override all levels of config. Some config entries can only be specified by means of CLI params.
Levels Priority Source Description Location
Default Lowest Default Default values for code logic definitions roles/<role>/defaults/main.yml
Global Low Global Consistent within the same set of deployment envs all.vars.xxx
Cluster Medium Cluster Consistency within the same set of clusters all.children.<cls>.vars.xxx
Instance High Instance The most granular level of config all.children.<cls>.hosts.<ins>.xxx
Argument Highest Parameter Pass in CLI arguments -e

Config Category

The v1.5.1 source-backed summary contains 213 fixed config entries divided into four sections: INFRA, NODES, PGSQL, and REDIS, for a total of 32 categories.

Usually, only the node/database identity parameter is mandatory, other params can be modified on demand using the default values.

Category Section Description Count
INFRA CONNECT Connection parameters 1
INFRA REPO Local source infra 10
INFRA CA Public-Private Key Infra 5
INFRA NGINX Nginx Web Server 5
INFRA NAMESERVER DNS Server 1
INFRA PROMETHEUS Monitoring Time Series Database 7
INFRA EXPORTER Universal Exporter Config 3
INFRA GRAFANA Grafana Visualization Platform 9
INFRA LOKI Loki log collection platform 5
INFRA DCS Distributed Config Storage Meta DB 8
NODES NODE_IDENTITY Node identity parameters 5
NODES NODE_DNS Node Domain Name Resolution 5
NODES NODE_PACKAGES Node Packages 4
NODES NODE_KERNEL_MODULES Node Kernel Module 1
NODES NODE_TUNE Node parameter tuning 2
NODES NODE_ADMIN Node Admin User 6
NODES NODE_TIME Node time zone and time sync 4
NODES NODE_EXPORTER Node Indicator Exposer 3
NODES PROMTAIL Log collection component 5
PGSQL PG_IDENTITY PGSQL Identity Parameters 13
PGSQL PG_BUSINESS PGSQL Business Object Definition 11
PGSQL PG_INSTALL PGSQL Installation 11
PGSQL PG_BOOTSTRAP PGSQL Cluster Initialization 24
PGSQL PG_PROVISION PGSQL Cluster Provisioning 9
PGSQL PG_EXPORTER PGSQL Indicator Exposer 13
PGSQL PG_SERVICE PGSQL Service Access 16
REDIS REDIS_IDENTITY REDIS Identity Parameters 3
REDIS REDIS_PROVISION REDIS Cluster Provisioning 14
REDIS REDIS_NODE REDIS Indicator Exposer 3
List of config entries
ID Name Section Level Description
100 proxy_env CONNECT G Proxy server config
110 nginx_enabled REPO G Enable local sources
111 repo_name REPO G Local source name
112 repo_address REPO G Local source external access address
113 nginx_port REPO G Local source port
114 nginx_home REPO G Local source file root dir
115 repo_rebuild REPO A Rebuild Yum repo
116 repo_remove REPO A Remove existing REPO files
117 repo_upstreams REPO G Upstream sources of Yum repo
118 repo_packages REPO G List of software from Yum repo
119 repo_url_packages REPO G List of software downloaded via URL
120 ca_method CA G CA creation method
121 ca_subject CA G Self-signed CA themes
122 ca_homedir CA G CA root dir
123 ca_cert CA G CA Certificate
124 ca_key CA G CA private key name
130 nginx_upstream NGINX G Nginx upstream servers
131 nginx_indexes NGINX G List of apps displayed on the navigation bar
140 dns_records NAMESERVER G Dynamic DNS Resolution Records
150 prometheus_data_dir PROMETHEUS G Prometheus Catalog
151 prometheus_options PROMETHEUS G Prometheus CLI parameters
152 prometheus_reload PROMETHEUS A Reload instead of Recreate
153 prometheus_sd_method PROMETHEUS G Service discovery mechanism: static
154 prometheus_scrape_interval PROMETHEUS G Prom Crawl Cycle
155 prometheus_scrape_timeout PROMETHEUS G Prom Crawl Timeout
156 prometheus_sd_interval PROMETHEUS G Prom Service Discovery Refresh Cycle
160 exporter_install EXPORTER G Installation of monitoring components method
161 exporter_repo_url EXPORTER G YumRepo for monitoring components
162 exporter_metrics_path EXPORTER G Monitor the exposed URL Path
170 grafana_endpoint GRAFANA G Grafana Address
171 grafana_admin_username GRAFANA G Grafana Admin Username
172 grafana_admin_password GRAFANA G Grafana Admin User Password
173 grafana_database GRAFANA G Grafana Database Types
174 grafana_pgurl GRAFANA G Grafana’s PG connection string
175 grafana_plugin_method GRAFANA G Grafana plugin installation method
176 grafana_plugin_cache GRAFANA G Grafana plugin cache location
177 grafana_plugin_list GRAFANA G Installation list of Grafana plugins
178 grafana_plugin_git GRAFANA G Installing Grafana Plugin from Git
180 loki_endpoint LOKI G Receiving logs for the loki service
181 loki_clean LOKI A Clean up the database dir during Loki installation
182 loki_options LOKI G Loki’s CLI parameters
183 loki_data_dir LOKI G Loki’s data dir
184 loki_retention LOKI G Loki log default retention days
200 dcs_servers DCS G DCS server name:IP list
201 dcs_registry DCS G Service Registration Location
202 pg_dcs_type DCS G DCS Type
203 dcs_name DCS G DCS Cluster Name
204 dcs_clean DCS C/A Action when DCS instance exists
205 dcs_safeguard DCS C/A Prohibit cleaning of DCS instances
206 consul_data_dir DCS G Consul Data Catalog
207 etcd_data_dir DCS G Etcd Data Catalog
221 jupyter_username JUPYTER G OS users used by Jupyter
222 jupyter_password JUPYTER G Jupyter Lab Password
300 meta_node NODE_IDENTITY C Meta Node
301 nodename NODE_IDENTITY I Node instance mark
302 node_cluster NODE_IDENTITY C Node cluster name, default nodes
303 nodename_overwrite NODE_IDENTITY C Nodename overrides HOSTNAME
304 nodename_exchange NODE_IDENTITY C Exchange hostnames between playbook nodes
310 node_etc_hosts_default NODE_DNS C Static DNS Analysis
311 node_etc_hosts NODE_DNS C/I Cluster Level
312 node_dns_method NODE_DNS C Configure DNS server method
313 node_dns_servers NODE_DNS C Configure a list of dynamic DNS servers
314 node_dns_options NODE_DNS C Configure the /etc/resolv.conf
320 node_repo_method NODE_REPO C The way nodes use Yum repos
321 node_repo_remove NODE_REPO C Remove nodes with existing Yum repos
322 node_repo_local_urls NODE_REPO C URL of the local source
330 node_packages_default NODE_PACKAGES C Packages for nodes
331 node_packages NODE_PACKAGES C Extra packages for nodes
332 node_packages_meta NODE_PACKAGES G Packages for meta nodes
333 node_packages_meta_pip NODE_PACKAGES G Packages installed via pip3
340 node_disable_numa NODE_FEATURES C Disable the node NUMA
341 node_disable_swap NODE_FEATURES C Disable the node SWAP
342 node_disable_firewall NODE_FEATURES C Disable the node firewall
343 node_disable_selinux NODE_FEATURES C Disable the node SELINUX
344 node_static_network NODE_FEATURES C Enable static DNS servers
345 node_disk_prefetch NODE_FEATURES C Enable disk pre-reading
346 node_kernel_modules NODE_MODULES C Enable kernel module
350 node_tune NODE_TUNE C Node Tuning Mode
351 node_sysctl_params NODE_TUNE C OS kernel parameters
360 node_admin_enabled NODE_ADMIN G Create admin user
361 node_admin_uid NODE_ADMIN G Admin UID
362 node_admin_username NODE_ADMIN G Admin User Name
363 node_admin_ssh_exchange NODE_ADMIN C Exchange admin user SSH keys
364 node_admin_pk_current NODE_ADMIN A Add the current user’s public key to the admin user
365 node_admin_pk_list NODE_ADMIN C Login admin’s public key list
370 node_timezone NODE_TIME C NTP time zone setting
371 node_ntp_enabled NODE_TIME C Configure NTP service
372 node_ntp_service NODE_TIME C NTP service type: ntp or chrony
373 node_ntp_servers NODE_TIME C NTP Server List
380 node_exporter_enabled NODE_EXPORTER C Enable node metrics collector
381 node_exporter_port NODE_EXPORTER C Node Indicator Exposure Port
382 node_exporter_options NODE_EXPORTER C/I Node Metrics Collection Options
390 promtail_enabled PROMTAIL C Enable Protail log collection
391 promtail_clean PROMTAIL C/A Remove existing status information when installing promtail
392 promtail_port PROMTAIL G promtail default port
393 promtail_options PROMTAIL C/I promtail CLI parameters
394 promtail_positions PROMTAIL C promtail status file location
500 pg_cluster PG_IDENTITY C PG database cluster name
501 pg_shard PG_IDENTITY C PG Cluster-owned Shard (Reserved)
502 pg_sindex PG_IDENTITY C PG cluster’s slice number (Reserved)
503 gp_role PG_IDENTITY C Role of PG Cluster in GP
504 pg_role PG_IDENTITY I PG instance role
505 pg_seq PG_IDENTITY I PG Instance Serial Number
506 pg_instances PG_IDENTITY I All PG instances on the current node
507 pg_upstream PG_IDENTITY I Replicated upstream nodes of instances
508 pg_offline_query PG_IDENTITY I Offline Search
509 pg_backup PG_IDENTITY I Storing backups on instances
510 pg_weight PG_IDENTITY I Relative weight of instances in load balancing
511 pg_hostname PG_IDENTITY C/I PG instance name is set to HOSTNAME
512 pg_preflight_skip PG_IDENTITY C/A Skip PG identity parameter checks
520 pg_users PG_BUSINESS C Business User Definition
521 pg_databases PG_BUSINESS C Business Database Definition
522 pg_services_extra PG_BUSINESS C Cluster Proprietary Services Definition
523 pg_hba_rules_extra PG_BUSINESS C Cluster/instance specific HBA rules
524 pgbouncer_hba_rules_extra PG_BUSINESS C Pgbounce specific HBA rules
525 pg_admin_username PG_BUSINESS G PG Admin Users
526 pg_admin_password PG_BUSINESS G PG Adimin User Passwords
527 pg_replication_username PG_BUSINESS G PG Replica User
528 pg_replication_password PG_BUSINESS G PG Replica User Passwords
529 pg_monitor_username PG_BUSINESS G PG Monitor Users
530 pg_monitor_password PG_BUSINESS G PG Monitor User passwords
540 pg_dbsu PG_INSTALL C PG OS Super User
541 pg_dbsu_uid PG_INSTALL C Super UID
542 pg_dbsu_sudo PG_INSTALL C Sudo privileges for super users
543 pg_dbsu_home PG_INSTALL C Super User’s Home Dir
544 pg_dbsu_ssh_exchange PG_INSTALL C Exchanging Super User Keys
545 pg_version PG_INSTALL C Large version of the installed database
546 pgdg_repo PG_INSTALL C Add PG official repo
547 pg_add_repo PG_INSTALL C Add PG-related upstream repos
548 pg_bin_dir PG_INSTALL C PG Binary Dir
549 pg_packages PG_INSTALL C List of installed PG packages
550 pg_extensions PG_INSTALL C List of installed PG plug-ins
560 pg_clean PG_BOOTSTRAP C/A Handling method when PG exists
561 pg_safeguard PG_BOOTSTRAP C/A Prohibit clearing of existing PG instances
562 pg_data PG_BOOTSTRAP C PG data dir
563 pg_fs_main PG_BOOTSTRAP C PG master data disk mount point
564 pg_fs_bkup PG_BOOTSTRAP C PG backup disk mount point
565 pg_dummy_filesize PG_BOOTSTRAP C The size of the placeholder file /pg/dummy
566 pg_listen PG_BOOTSTRAP C PG listening IP
567 pg_port PG_BOOTSTRAP C PG listening port
568 pg_localhost PG_BOOTSTRAP C UnixSocket address used by PG
580 patroni_enabled PG_BOOTSTRAP C Enable Patroni
581 patroni_mode PG_BOOTSTRAP C Patroni config mode
582 pg_namespace PG_BOOTSTRAP C Patroni’s DCS namespace
583 patroni_port PG_BOOTSTRAP C Patroni service port
584 patroni_watchdog_mode PG_BOOTSTRAP C Patroni Watchdog mode
585 pg_conf PG_BOOTSTRAP C Patroni’s config templates
586 pg_libs PG_BOOTSTRAP C PG Default Shared database
587 pg_encoding PG_BOOTSTRAP C PG character set encoding
588 pg_locale PG_BOOTSTRAP C Localization rules for PG
589 pg_lc_collate PG_BOOTSTRAP C Localized sorting rules for PG
590 pg_lc_ctype PG_BOOTSTRAP C Localized character set definitions for PG
591 pgbouncer_enabled PG_BOOTSTRAP C Enable Pgbouncer
592 pgbouncer_port PG_BOOTSTRAP C Pgbouncer Port
593 pgbouncer_poolmode PG_BOOTSTRAP C Pgbouncer pooling mode
594 pgbouncer_max_db_conn PG_BOOTSTRAP C Pgbouncer Maximum DB connections
600 pg_provision PG_PROVISION C Applying templates in PG clusters
601 pg_init PG_PROVISION C Custom PG initialization script
602 pg_default_roles PG_PROVISION G/C Default Roles and Users
603 pg_default_privileges PG_PROVISION G/C Database default privileges config
604 pg_default_schemas PG_PROVISION G/C Default Mode
605 pg_default_extensions PG_PROVISION G/C Extensions installed by default
606 pg_reload PG_PROVISION A Reload Database Config (HBA)
607 pg_hba_rules PG_PROVISION G/C Global HBA rules
608 pgbouncer_hba_rules PG_PROVISION G/C Pgbouncer Global HBA rules
620 pg_exporter_config PG_EXPORTER C PG Metrics Definition Document
621 pg_exporter_enabled PG_EXPORTER C Enable PG Indicator Collector
622 pg_exporter_port PG_EXPORTER C PG Indicator Exposure Port
623 pg_exporter_params PG_EXPORTER C/I Extra URL parameters for PG Exporter
624 pg_exporter_url PG_EXPORTER C/I Acquisition of connection strings for object databases (override)
625 pg_exporter_auto_discovery PG_EXPORTER C/I Auto-discovery of the database in the instance
626 pg_exporter_exclude_database PG_EXPORTER C/I Automatic database exclusion list
627 pg_exporter_include_database PG_EXPORTER C/I Automatic database capsule list
628 pg_exporter_options PG_EXPORTER C/I PG Exporter CLI parameters
629 pgbouncer_exporter_enabled PG_EXPORTER C Enable PGB Indicator Collector
630 pgbouncer_exporter_port PG_EXPORTER C PGB Indicator Exposure Port
631 pgbouncer_exporter_url PG_EXPORTER C/I Collection of connection strings for object connection pools
632 pgbouncer_exporter_options PG_EXPORTER C/I PGB Exporter CLI Parameters
640 pg_services PG_SERVICE G/C Global Common Service Definition
641 haproxy_enabled PG_SERVICE C/I Enable Haproxy
642 haproxy_reload PG_SERVICE A Reload Haproxy config
643 haproxy_auth_enabled PG_SERVICE G/C Enable authentication for the Haproxy management interface
644 haproxy_admin_username PG_SERVICE G HAproxy admin user name
645 haproxy_admin_password PG_SERVICE G HAproxy admin user password
646 haproxy_exporter_port PG_SERVICE C HAproxy metrics exposer port
647 haproxy_client_timeout PG_SERVICE C HAproxy client timeout
648 haproxy_server_timeout PG_SERVICE C HAproxy server timeout
649 vip_mode PG_SERVICE C VIP mode：none
650 vip_reload PG_SERVICE A Overload VIP Config
651 vip_address PG_SERVICE C The cluster’s VIP address
652 vip_cidrmask PG_SERVICE C Network CIDR mask length for VIP address
653 vip_interface PG_SERVICE C VIP’s network card
654 dns_mode PG_SERVICE C DNS config mode
655 dns_selector PG_SERVICE C DNS Object Selector
700 redis_cluster REDIS_IDENTITY C Redis Cluster Name
701 redis_node REDIS_IDENTITY I Redis Node Serial Number
702 redis_instances REDIS_IDENTITY I Redis Instance Definition
721 redis_mode REDIS_PROVISION C Redis Cluster Mode
722 redis_conf REDIS_PROVISION C Redis Config Template
723 redis_fs_main REDIS_PROVISION C PG Instance Role
724 redis_bind_address REDIS_PROVISION C Redis listening port
725 redis_clean REDIS_PROVISION C Actions when Redis exists
726 redis_safeguard REDIS_PROVISION C Disable wiping of existing Redis
727 redis_max_memory REDIS_PROVISION C/I Maximum memory available to Redis
728 redis_mem_policy REDIS_PROVISION C Memory Eviction Policy
729 redis_password REDIS_PROVISION C Redis passwords
730 redis_rdb_save REDIS_PROVISION C RDB save command
731 redis_aof_enabled REDIS_PROVISION C Enable AOF
732 redis_rename_commands REDIS_PROVISION C Rename Danger Command List
740 redis_cluster_replicas REDIS_PROVISION C Each master with several slaves
741 redis_exporter_enabled REDIS_NODE C Enabling Redis Monitoring
742 redis_exporter_port REDIS_NODE C Redis Exporter Listening Port
743 redis_exporter_options REDIS_NODE C/I Redis Exporter Command Parameters

27 - Config: Infra

Historical documentation restored from the Pigsty v1.5.1 tag.

Use the INFRA Playbook, and deploy the PGSQL cluster to adjust the cluster state to the state described in PGSQL Config.

Use the INFRA series playbooks to configure the Pigsty infra.

Infra config deals with such issues: localYum repos, machine node base services: DNS, NTP, kernel modules, parameter tuning, admin users, installing packages, DCS Server setup, monitor infra installation, and initialization (Grafana, Prometheus, Alertmanager), global traffic portal Nginx config, etc.

Usually, the infra requires very few modifications, and the main modification is just a text replacement of the meta node IPs, which is done in ./configure automatically. The other occasional change is to the access domain defined in nginx_upstream. Other parameters are adjusted as needed.

  • CONNECT: Connection parameters
  • CA: CA PKI Infra
  • NGINX: Nginx Web Server
  • REPO: Local repo infra
  • NAMESERVER: DNS Server
  • PROMETHEUS: Monitor Time Series Database
  • EXPORTER: Universal Exporter Config
  • GRAFANA : Grafana Visualization Platform
  • LOKI: Loki log collection platform
  • DCS: Distributed Configure Storage Meta DB
  • CONSUL: DCS Implementation: Consul
  • ETCD: DCS Implementation: ETCD

Parameter Overview

The following config entries describe the infra deployed on the meta node.

ID Name Section Type Level Comment
100 proxy_env CONNECT dict G proxy env variables
110 ca_method CA enum G ca mode, create,copy,recreate
111 ca_subject CA string G ca subject
112 ca_homedir CA path G ca cert home dir
113 ca_cert CA string G ca cert file name
114 ca_key CA string G ca private key name
120 nginx_enabled NGINX bool C/I enable nginx web server
121 nginx_home NGINX path G nginx home dir (/www)
122 nginx_port NGINX int G nginx listen address (80)
123 nginx_upstream NGINX upstream[] G nginx upstream definition
124 nginx_indexes NGINX app[] G nginx index page nav entries
130 repo_name REPO string G local yum repo name
131 repo_address REPO string G external access port of repo
132 repo_rebuild REPO bool A rebuild local yum repo
133 repo_remove REPO bool A remove existing repo file
134 repo_upstreams REPO repo[] G upstream repo definition
135 repo_packages REPO string[] G packages to be downloaded
136 repo_url_packages REPO url[] G pkgs to be downloaded via url
140 nameserver_enabled NAMESERVER bool C/I enable dnsmasq on meta node
141 dns_records NAMESERVER string[] G dynamic DNS records
150 prometheus_enabled PROMETHEUS bool C/I enable Prometheus on meta
151 prometheus_data_dir PROMETHEUS path G prometheus data dir
152 prometheus_options PROMETHEUS string G prometheus cli args
153 prometheus_reload PROMETHEUS bool A prom reload instead of init
154 prometheus_sd_method PROMETHEUS enum G consul
155 prometheus_scrape_interval PROMETHEUS interval G prom scrape interval (10s)
156 prometheus_scrape_timeout PROMETHEUS interval G prom scrape timeout (8s)
157 prometheus_sd_interval PROMETHEUS interval G prom discovery refresh interval
160 exporter_install EXPORTER enum G Installation of exporter
161 exporter_repo_url EXPORTER string G repo url for yum install
162 exporter_metrics_path EXPORTER string G URL path for exporting metrics
170 grafana_enabled GRAFANA bool C/I enable grafana on meta node
171 grafana_endpoint GRAFANA url G grafana API endpoint
172 grafana_admin_username GRAFANA string G grafana admin username
173 grafana_admin_password GRAFANA string G grafana admin password
174 grafana_database GRAFANA enum G grafana backend database type
175 grafana_pgurl GRAFANA url G grafana backend postgres url
176 grafana_plugin_method GRAFANA enum G Install grafana plugin method
177 grafana_plugin_cache GRAFANA path G grafana plugins cache path
178 grafana_plugin_list GRAFANA string[] G grafana plugins to be installed
179 grafana_plugin_git GRAFANA url[] G grafana plugins via git
180 loki_enabled LOKI bool C/I enable loki on meta node
180 loki_endpoint LOKI url G loki endpoint to receive log
181 loki_clean LOKI bool A remove existing loki data
182 loki_options LOKI string G loki cli args
183 loki_data_dir LOKI string G loki data path
184 loki_retention LOKI interval G loki log keeping period
190 dcs_name DCS string G dcs cluster name (dc)
191 dcs_servers DCS dict G dcs server dict
192 dcs_registry DCS enum G Registration Services
193 dcs_safeguard DCS bool C/A avoid dcs remove at all
194 dcs_clean DCS bool C/A purge dcs during init?
195 consul_enabled CONSUL bool G enable consul servers/agents
196 consul_data_dir CONSUL string G consul data dir path
197 etcd_enabled ETCD bool G enable etcd servers/clients
198 etcd_data_dir ETCD string G etcd data dir path

CONNECT

proxy_env

Using a proper HTTP proxy, download speeds of several MB per second can be achieved. If you have a proxy server, please configure it via proxy_env. The sample example is as follows.

proxy_env: # global proxy env when downloading packages
  http_proxy: 'http://username:[email protected]'
  https_proxy: 'http://username:[email protected]'
  all_proxy: 'http://username:[email protected]'
  no_proxy: "localhost,127.0.0.1,10.0.0.0/8,192.168.0.0/16,*.pigsty,*.aliyun.com,mirrors.aliyuncs.com,mirrors.tuna.tsinghua.edu.cn,mirrors.zju.edu.cn"

ansible_host

If considering using the Ansible connection parameter, your target machine is hidden behind an SSH springboard machine or is not accessible via ssh ip.

For example, in the example below, ansible_host tells Pigsty to access the target database node using an SSH alias using the ssh node-1 method instead of the ssh 10.10.10.11 method. This allows you to freely specify the connection method of the database node and save the connection configuration in the ~/.ssh/config of the admin user for independent management.

  pg-test:
    vars: { pg_cluster: pg-test }
    hosts:
      10.10.10.11: {pg_seq: 1, pg_role: primary, ansible_host: node-1}
      10.10.10.12: {pg_seq: 2, pg_role: replica, ansible_host: node-2}
      10.10.10.13: {pg_seq: 3, pg_role: offline, ansible_host: node-3}

ansible_host is the most typical of the ansible connection parameters. Usually, as long as the user can access the target machine via ssh <name>, configuring the ansible_host variable, for instance, with a value of <name> and other common Ansible SSH connection parameters are shown below:

  • ansible_host: Specify the target machine’s IP, hostname, or SSH alias.

  • ansible_port: Specify a different SSH port than 22

  • ansible_user: Specify the username to use for SSH

  • ansible_ssh_pass: SSH password (Do not store plaintext, and input from the keyboard can be specified by the -k)

  • ansible_ssh_private_key_file: SSH private key path

  • ansible_ssh_common_args: SSH General Parameters

CA

Self-Signed CA PKI.

ca_method

CA creation method, type: enum, level: G, default value: "create".

  • create： Create a new CA public-private key pair.
  • copy： Copy the existing CA public and private keys for building CA.

ca_subject

Self-signed CA theme, type: string, level: G, default value: "/CN=root-ca".

ca_homedir

CA certificate root dir, type: path, level: G, default value: "/ca".

ca_cert

CA certificate, type: string, level: G, default value: "ca.crt".

ca_key

CA private key name, type: string, level: G, default value: "ca.key".

NGINX

Pigsty exposes all Web services through Nginx: Home Page, Grafana, Prometheus, AlertManager, Consul, and other optional tools such as PGWe, Jupyter Lab, Pgadmin, Bytebase ,and other static resource & report such as pgweb schemaspy & pgbadger

Some services on the meta node can be accessed directly through the port, bypassing Nginx, but some services can only be accessed through the Nginx proxy for security reasons. Nginx distinguishes between different services by the domain name. If the domain name configured for each service does not resolve in the current environment, you will need to configure it in /etc/hosts.

nginx_enabled

Enable nginx (and yum repo), type: bool, level: C/I, default value: true.

Setup nginx server on current meta node?

Set to false will skip it.

You can set this parameter to false on standby meta nodes when using multiple meta nodes

nginx_port

Local repo port, type: int, level: G, default value: 80.

Pigsty accesses all web services through this port on the meta node. Make sure you can access this port on the meta node.

nginx_home

Local repo root, type: path, level: G, default value: "/www".

Nginx root directory which contains static resource and repo resource.

nginx_upstream

Nginx upstream server, Type: upstream[], Level: G, default value:

nginx_upstream:                  # domain names and upstream servers
  - { name: home,         domain: pigsty,     endpoint: "10.10.10.10:80" }
  - { name: grafana,      domain: g.pigsty,   endpoint: "10.10.10.10:3000" }
  - { name: loki,         domain: l.pigsty,   endpoint: "10.10.10.10:3100" }
  - { name: prometheus,   domain: p.pigsty,   endpoint: "10.10.10.10:9090" }
  - { name: alertmanager, domain: a.pigsty,   endpoint: "10.10.10.10:9093" }
  - { name: consul,       domain: c.pigsty,   endpoint: "127.0.0.1:8500" }
  - { name: pgweb,        domain: cli.pigsty, endpoint: "127.0.0.1:8081" }
  - { name: jupyter,      domain: lab.pigsty, endpoint: "127.0.0.1:8888" }

Each record contains three subsections: name, domain, and endpoint, representing the component name, the external access domain, and the internal TCP port, respectively.

The name definition of the default record is fixed and referenced by hard-coding, do not modify it. Upstream server records with other names can be added at will.

The domain is the domain name that should be used for external access to this upstream server. When accessing the Pigsty Web service, the domain name should be used to access it through the Nginx proxy.

The endpoint is an internally reachable TCP port. During the Configure, the placeholder IP 10.10.10.10 will be replaced with the meta node IP.

nginx_indexes

List of applications displayed in the home navigation bar, type: app[], level: G, default value:

nginx_indexes:                            # application nav links on home page
  - { name: Pev2    , url : '/pev2'        , comment: 'postgres explain visualizer 2' }
  - { name: Logs    , url : '/logs'        , comment: 'realtime pgbadger log sample' }
  - { name: Report  , url : '/report'      , comment: 'daily log summary report ' }
  - { name: Pkgs    , url : '/pigsty'      , comment: 'local yum repo packages' }
  - { name: Repo    , url : '/pigsty.repo' , comment: 'local yum repo file' }
  - { name: ISD     , url : '${grafana}/d/isd-overview'   , comment: 'noaa isd data visualization' }
  - { name: Covid   , url : '${grafana}/d/covid-overview' , comment: 'covid data visualization' }

Each record is rendered as a navigation link to the Pigsty home page App drop-down menu, and the apps are all optional, mounted by default on the Pigsty default server under http://pigsty/. The url parameter specifies the URL PATH for the app, with the exception that if the ${grafana} string is present in the URL, it will be automatically replaced with the Grafana domain name defined in nginx_upstream.

REPO

Pigsty is installed on a meta node. Pigsty pulls up a localYum repo for the current environment to install RPM packages.

During initialization, Pigsty downloads all packages and their dependencies (specified by repo_packages) from the Internet upstream repo (specified by repo_upstreams) to {{ nginx_home }} / {{ repo_name }} (default is /www/pigsty). The total size of all dependent software is about 1GB or so.

When creating a localYum repo, Pigsty will skip the software download phase if the directory already exists and if there is a marker file named repo_complete in the dir.

If the download speed of some packages is too slow, you can set the download proxy to complete the first download by using the proxy_env config entry or directly download the pre-packaged offline package.

The offline package is a zip archive of the {{ nginx_home }}/{{ repo_name }} dir pkg.tgz. During configure, if Pigsty finds the offline package /tmp/pkg.tgz, it will extract it to {{ nginx_home }}/{{ repo_name }}, skipping the software download step during installation.

The default offline package is based on CentOS 7.8.2003 x86_64; if you use a different OS, there may be RPM package conflict and dependency error problems; please refer to the FAQ to solve.

repo_name

Local repo name, type: string, level: G, default value: "pigsty". It is not recommended to modify this parameter.

repo_address

Local repo external access address, type: string, level: G, default value: "pigsty".

The address of the local yum repo for external services, either a domain name or an IP, the default is yum. pigsty.

If you use a domain name, you must ensure that the domain name will resolve correctly to the server where the local repo is located, i.e., the meta node.

If the local yum repo does not use the standard port 80, you need to add the port to the address and keep it consistent with the nginx_port variable.

The static DNS config node_etc_hosts_default in the nodes parameter can be used to write the pigsty local repo domain name by default for all nodes in the current env.

repo_rebuild

Rebuild Yum repo, type: bool, level: A, default value: false.

If true, then the Repo rebuild will be performed in all cases, i.e., regardless of whether the offline package exists.

repo_remove

Remove existing REPO files, type: bool, level: A, default value: true.

If true, the existing repo in /etc/yum.repos.d on the meta node will be removed and backed up to the /etc/yum.repos.d/backup dir during the local repo initialization process.

Since the content of existing reports in the OS is not controllable, it is recommended to force the removal of existing repos and configure them explicitly via repo_upstreams.

When the node has other self-configured repos or needs to download some particular version of RPM packages from a specific repo, it can be set to false to keep the existing repos.

repo_upstreams

Upstream source of Yum repo, type: repo[], level: G.

We use AliCloud’s CentOS7 mirror repo, Tsinghua University’s Grafana mirror repo, PackageCloud’s Prometheus repo, PostgreSQL official repo, and software repos such as SCLo, Harbottle, and Nginx.

repo_packages

List of software to download for Yum repo, type: string[], level: G, default value.

epel-release nginx wget yum-utils yum createrepo sshpass zip unzip                                              # ----  boot   ---- #
ntp chrony uuid lz4 bzip2 nc pv jq vim-enhanced make patch bash lsof wget git tuned perf ftp lrzsz rsync        # ----  node   ---- #
numactl grubby sysstat dstat iotop bind-utils net-tools tcpdump socat ipvsadm telnet ca-certificates keepalived # ----- utils ----- #
readline zlib openssl openssh-clients libyaml libxml2 libxslt libevent perl perl-devel perl-ExtUtils*           # ---  deps:pg  --- #
readline-devel zlib-devel uuid-devel libuuid-devel libxml2-devel libxslt-devel openssl-devel libicu-devel       # --- deps:devel -- #
grafana prometheus2 pushgateway alertmanager mtail consul consul_exporter consul-template etcd dnsmasq          # -----  meta ----- #
node_exporter nginx_exporter blackbox_exporter redis_exporter                                                   # ---- exporter --- #
ansible python python-pip python-psycopg2                                                                       # - ansible & py3 - #
python3 python3-psycopg2 python36-requests python3-etcd python3-consul python36-urllib3 python36-idna python36-pyOpenSSL python36-cryptography
patroni patroni-consul patroni-etcd pgbouncer pg_cli pgbadger pg_activity tail_n_mail                           # -- pgsql common - #
pgcenter boxinfo check_postgres emaj pgbconsole pg_bloat_check pgquarrel barman barman-cli pgloader pgFormatter pitrery pspg pgxnclient PyGreSQL
postgresql14* postgis32_14* citus_14* pglogical_14* timescaledb-2-postgresql-14 pg_repack_14 wal2json_14        # -- pg14 packages -#
pg_qualstats_14 pg_stat_kcache_14 pg_stat_monitor_14 pg_top_14 pg_track_settings_14 pg_wait_sampling_14 pg_probackup-std-14
pg_statement_rollback_14 system_stats_14 plproxy_14 plsh_14 pldebugger_14 plpgsql_check_14 pgmemcache_14 # plr_14
mysql_fdw_14 ogr_fdw_14 tds_fdw_14 sqlite_fdw_14 firebird_fdw_14 hdfs_fdw_14 mongo_fdw_14 osm_fdw_14 pgbouncer_fdw_14
hypopg_14 geoip_14 rum_14 hll_14 ip4r_14 prefix_14 pguri_14 tdigest_14 topn_14 periods_14
bgw_replstatus_14 count_distinct_14 credcheck_14 ddlx_14 extra_window_functions_14 logerrors_14 mysqlcompat_14 orafce_14
repmgr_14 pg_auth_mon_14 pg_auto_failover_14 pg_background_14 pg_bulkload_14 pg_catcheck_14 pg_comparator_14
pg_cron_14 pg_fkpart_14 pg_jobmon_14 pg_partman_14 pg_permissions_14 pg_prioritize_14 pgagent_14
pgaudit16_14 pgauditlogtofile_14 pgcryptokey_14 pgexportdoc_14 pgfincore_14 pgimportdoc_14 powa_14 pgmp_14 pgq_14
pgquarrel-0.7.0-1 pgsql_tweaks_14 pgtap_14 pgtt_14 postgresql-unit_14 postgresql_anonymizer_14 postgresql_faker_14
safeupdate_14 semver_14 set_user_14 sslutils_14 table_version_14 # pgrouting_14 osm2pgrouting_14
clang coreutils diffutils rpm-build rpm-devel rpmlint rpmdevtools bison flex # gcc gcc-c++                      # - build utils - #
docker-ce docker-compose kubelet kubectl kubeadm kubernetes-cni helm                                            # - cloud native- #

Each line is a set of package names separated by spaces, where the specified software will be downloaded via repotrack.

repo_url_packages

Software for direct download via URL, type: url[], level: G

Download some software via URL, not YUM:

  • pg_exporter: Must, core components of the monitor system.
  • vip-manager: Must, package required to enable L2 VIP for managing VIP.
  • loki, promtail: Must, log collection server-side and client-side binary.
  • postgrest: Optional, automatically generate backend API interface based on PostgreSQL mode.
  • polysh: Optional, execute ssh commands on multiple nodes in parallel.
  • pev2: Optional, PostgreSQL execution plan visualization
  • pgweb: Optional, web-based PostgreSQL CLI tool
  • redis: Optional, mandatory when Redis is installed
https://github.com/Vonng/loki-rpm/releases/download/v2.5.0/loki-2.5.0.x86_64.rpm
https://github.com/Vonng/loki-rpm/releases/download/v2.5.0/promtail-2.5.0.x86_64.rpm
https://github.com/Vonng/pg_exporter/releases/download/v0.5.0/pg_exporter-0.5.0.x86_64.rpm
https://github.com/cybertec-postgresql/vip-manager/releases/download/v1.0.2/vip-manager-1.0.2-1.x86_64.rpm
https://github.com/Vonng/haproxy-rpm/releases/download/v2.6.0/haproxy-2.6.0-1.el7.x86_64.rpm
https://github.com/Vonng/pigsty-pkg/releases/download/misc/redis-6.2.7-1.el7.remi.x86_64.rpm
https://github.com/dalibo/pev2/releases/download/v0.24.0/pev2.tar.gz
https://github.com/Vonng/pigsty-pkg/releases/download/misc/polysh-0.4-1.noarch.rpm

NAMESERVER

Pigsty will default use DNSMASQ to build an optional battery-included name server on the meta node.

nameserver_enabled

Enable DNSMASQ on the meta node, type: bool, level: C/I, default value: false.

dns_records

Dynamic DNS resolution record, type: string[], level: G, default value is [] empty list, the following resolution records are available by default in the sandbox.

dns_records:                    # dynamic dns record resolved by dnsmasq
  - 10.10.10.2  pg-meta         # sandbox vip for pg-meta
  - 10.10.10.3  pg-test         # sandbox vip for pg-test
  - 10.10.10.10 meta-1          # sandbox node meta-1
  - 10.10.10.11 node-1          # sandbox node node-1
  - 10.10.10.12 node-2          # sandbox node node-2
  - 10.10.10.13 node-3          # sandbox node node-3
  - 10.10.10.10 pg-meta-1       # sandbox instance pg-meta-1
  - 10.10.10.11 pg-test-1       # sandbox instance node-1
  - 10.10.10.12 pg-test-2       # sandbox instance node-2
  - 10.10.10.13 pg-test-3       # sandbox instance node-3

PROMETHEUS

Prometheus is the core component of the Pigsty monitor system, used to pull timing data, perform metrics precomputation, and evaluate alarm rules.

prometheus_enabled

Enable Prometheus on the meta node, type: bool, level: C/I, default value: true.

prometheus_data_dir

Prometheus dir, type: path, level: G, default value: "/data/prometheus/data".

prometheus_options

Prometheus CLI parameter, type: string, level: G, default value: "--storage.tsdb.retention=15d".

The default parameter retains monitoring data for 15 days. If you have a large enough disk, you can increase the length of time that monitoring data is kept.

prometheus_reload

Reload the configuration instead of rebuilding the whole thing when performing Prometheus tasks. Type: bool, Level: A, Default: false.

By default, executing the prometheus task will clear existing monitoring data, but if set to true, it will not.

prometheus_sd_method

Service discovery mechanism: static|consul, type: enum, level: G, default value: "static".

Prometheus’s service discovery mechanism, default static, option consul Use Consul for service discovery (will be phased out). Pigsty recommends using static for service discovery, which provides more excellent reliability and flexibility.

static service discovery relies on the config in /etc/prometheus/targets/{infra,nodes,pgsql,redis}/*.yml for service discovery.

The advantage of this method is that the monitoring system does not rely on consult. The monitoring target will give an error prompt when the node goes down instead of disappearing directly. In addition, when the pigsty monitor system is integrated with the external control mode, this mode is less invasive to the original system.

The following command can be used to generate the required monitoring object profile for Prometheus from the config file.

./nodes.yml -t register_prometheus  # Generate a list of host monitoring targets
./pgsql.yml -t register_prometheus  # Generate a list of PostgreSQL/Pgbouncer/Patroni/Haproxy monitoring targets
./redis.yml -t register_prometheus  # Generate a list of Redis monitoring targets

prometheus_scrape_interval

Prometheus crawl period, type: interval, level: G, default value: "10s".

Ten seconds - 30 seconds is a suitable crawl period. If a finer granularity of monitoring data is required, this parameter can be adjusted.

prometheus_scrape_timeout

Prometheus grab timeout, type: interval, level: G, default value: "8s".

Setting the crawl timeout can effectively avoid avalanches caused by monitoring system queries. This parameter must be less than and close to prometheus_scrape_interval to ensure that the length of each crawl does not exceed the crawling period.

prometheus_sd_interval

Prometheus service discovery refresh period, type: interval, level: G, default value: "5s".

Prometheus re-examines the local file dir every time specified by this parameter and refreshes the monitoring target object.

EXPORTER

Define generic metrics exporter options, such as how the Exporter is installed, the URL path to listen to, etc.

exporter_install

To install the monitoring component, type: enum, level: G, default value: "none".

Specify how to install Exporter:

  • none： No installation, (by default, the Exporter has been previously installed by the node.pkgs task)
  • yum： Install using yum (if yum installation is enabled, run yum to install node_exporter and pg_exporter before deploying Exporter)
  • binary： Install using a copy binary (copy node_exporter and pg_exporter binary directly from the meta node, not recommended)

When installing with yum, if exporter_repo_url is specified (not empty), the installation will first install the REPO file under that URL into /etc/yum.repos.d. This feature allows you to install Exporter directly without initializing the node infrastructure. It is not recommended for regular users to use binary installation. This mode is usually used for emergency troubleshooting and temporary problem fixes.

<meta>:<pigsty>/files/node_exporter ->  <target>:/usr/bin/node_exporter
<meta>:<pigsty>/files/pg_exporter   ->  <target>:/usr/bin/pg_exporter

exporter_repo_url

Yum Repo URL of the monitor component, type: string, level: G, default value: "".

Default is empty; when exporter_install is yum, the repo specified by this parameter will be added to the node source list.

exporter_metrics_path

Monitor the exposed URL Path, type: string, level: G, default value: "/metrics".

The URL PATH for all Exporter externally exposed metrics, which defaults to /metrics, is referenced by the external role prometheus, and Prometheus will apply this config to the monitoring object based on the config here.

Indicator exponents affected by this parameter include:

  • node_exporter
  • pg_exporter
  • pgbouncer_port
  • haproxy
  • Patroni’s Metrics endpoint is currently fixed to /metrics and cannot be configured, so it is not affected by this parameter.
  • The Metrics endpoint of the Infra component is fixed to /metrics and is not affected by this parameter.

GRAFANA

Grafana is the visualization platform for Pigsty’s monitoring system.

grafana_enabled

Enable Grafana on the meta node, type: bool, level: C/I, default value: true.

grafana_endpoint

Grafana address, type: url, level: G, default value: "http://10.10.10.10:3000".

Grafana provides a service endpoint to the public, which is used by the Grafana initialization and installation monitoring panel to call the Grafana API.

The placeholder IP 10.10.10.10 will be replaced by the actual IP during the configure.

grafana_admin_username

Grafana administrator username, type: string, level: G, default value: "admin".

grafana_admin_password

Grafana administrator password, type: string, level: G, default value: "pigsty".

grafana_database

Grafana backend database type, type: enum, tier: G, default value: "sqlite3".

The alternative is postgres. When using postgres, you must ensure that the target database already exists and is accessible. That is, Postgres on the meta node cannot be used before the initialization of the infrastructure for the first time because Grafana was created before that database.

To avoid creating circular dependencies (Grafana depends on Postgres, PostgreSQL depends on the infra, including Grafana), you need to modify this parameter and re-execute grafana-related tasks after the first time you complete the installation. For details, please see Tutorial: Using Postgres as a Grafana database.

grafana_pgurl

PostgreSQL connection string for Grafana, type: url, level: G, default value: "postgres://dbuser_grafana:DBUser.Grafana@meta:5436/grafana".

Only valid if the parameter grafana_database is postgres.

grafana_plugin_method

Install the Grafana plugin, type: enum, level: G, default value: "install".

How Grafana plug-ins are provisioned:

  • none： No plug-in installation.
  • install: Install the Grafana plugin (default), or skip it if it already exists.
  • reinstall: Re-download and install the Grafana plugin anyway.

Grafana requires Internet access to download several extension plug-ins, and if your meta-node does not have Internet access, you should ensure that you are using an offline installer. The offline installation package already contains all downloaded Grafana plugins by default, located under the path specified by grafana_plugin_cache. Pigsty will package the downloaded plugins and place them under that path after the download is complete when downloading plugins from the Internet.

grafana_plugin_cache

Grafana plugin cache address, type: path, level: G, default value: "/www/pigsty/plugins.tgz".

grafana_plugin_list

List of installed Grafana plugins, type: string[], level: G, default value:

grafana_plugin_list:
  - marcusolsson-csv-datasource
  - marcusolsson-json-datasource
  - marcusolsson-treemap-panel

Each array element is a string that represents the name of the plugin. Plugins are installed using grafana-cli plugins install.

grafana_plugin_git

Grafana plugin installed from Git, type: url[], level: G, default value:

grafana_plugin_git:                          # plugins that will be downloaded via git
  - https://github.com/Vonng/vonng-echarts-panel

Some plugins cannot be downloaded via the official command line but can be downloaded via Git Clone. Plugins will be installed via cd /var/lib/grafana/plugins && git clone .

A visualization plugin will be downloaded by default: vonng-echarts-panel, which provides Echarts drawing support for Grafana.

LOKI

LOKI is the default log collection server used by Pigsty.

loki_enabled

Enable Loki on the meta node, type: bool, level: C/I, default value: true.

loki_endpoint

Loki service endpoint for receiving logs, type: url, level: G, default value: "http://10.10.10.10:3100/loki/api/v1/push".

loki_clean

Clean up the database directory when installing Loki, type: bool, level: A, default value: false.

loki_options

Command line arguments for Loki, type: string, level: G, default value: "-config.file=/etc/loki.yml -config.expand-env=true".

The default config parameters are used to specify the Loki config file location and to enable the ability to expand environment variables in the config file; it is not recommended to remove these two options.

loki_data_dir

Loki’s data directory, type: string, level: G, default value: "/data/loki".

loki_retention

Loki log default retention days, type: interval, level: G, default value: "15d".

DCS

Distributed Configuration Store (DCS) is a distributed, highly available meta-database that provides HA consensus and service discovery.

Pigsty supports Consul & ETCD as DCS. Use dcs_registry to specify where to register service,

Availability of Consul/ETCD is critical for postgres HA. Special care needs to be taken when using the DCS service in a production env. Availability of DCS itself is achieved through multiple peers. For example, a 3-node DCS cluster allows up to one node to fail, while a 5-node DCS cluster allows 2 nodes to fail. In a large-scale production env, it is recommended to use at least 3~5 DCS Servers. The DCS servers used by Pigsty are specified by the parameter dcs_servers, either by using an existing external DCS server cluster or by deploying DCS Servers using nodes managed by Pigsty itself.

By default, Pigsty deploys setup DCS services when nodes are included in management (nodes.yml), and if the current node is defined in dcs_servers, the node will be initialized as a DCS Server. Pigsty deploys a single node DCS Server on the meta node itself by default. You can use any node as DCS Servers. Before deploying any HA Postgres Cluster, you should ensure that all DCS Servers have been initialized. (Which is done during nodes.yml)

dcs_name

DCS cluster name, type: string, level: G, default value: "pigsty".

Represents the data center name in Consul, and used as initial cluster token in Etcd.

dcs_servers

DCS Server, type: dict, level: G, default value:

dcs_servers:
  meta-1: 10.10.10.10      # Deploy a single DCS Server on the meta node by default
  # meta-2: 10.10.10.11
  # meta-3: 10.10.10.12

Key is the DCS server instance name, and Value is the server IP address. By default, Pigsty will configure the DCS service for the node in the node initialization playbook, which defaults to Consul.

You can use an external DCS server and fill in the addresses of all external DCS Servers. Otherwise, Pigsty will deploy a single instance DCS Server on the meta node (10.10.10.10 placeholder) by default. If the current node is defined in dcs_servers, i.e., the IP address matches any Value, the node will be initialized as a DCS Server, and its Key will be used as a Consul Server.

dcs_registry

Where to register service, type: enum, level: G, default value: "consul".

  • none： No service registration is performed (none will disable prometheus_sd_method = consul ).
  • consul： Registering services to Consul.
  • etcd： Registering services into Etcd (not supported yet).

pg_dcs_type

PostgreSQL HA DCS type, type: enum, level: G, default value: "consul".

The available v1.5.1 Patroni DCS backends are consul and etcd. The tagged Patroni and environment templates implement both; this is separate from the still-unimplemented dcs_registry=etcd service-registration option.

dcs_safeguard

Assure that any running consul instance will not be purged by any nodes playbook., level: C/A, default: false

Check SafeGuard for details.

dcs_clean

Remove an existing Consul instance during node initialization? The role fallback is true for Consul in v1.5.1 and the bundled pigsty.yml also sets dcs_clean: true; set it to false and/or enable dcs_safeguard for protected environments.

This allows the removal of any running consul instance during nodes.yml, which makes it a true idempotent playbook.

It’s a dangerous option so you’d better disable it by default and use it with -e CLI args.

Warning

This parameter not working when dcs_safeguard is set to true

Consul

Consul is used for service mesh, traffic control, health check, service registry, service discovery & consensus.

consul_enabled

Enable consul: bool, level: G, default value: true.

Setup consul servers & agents on all nodes.

consul_data_dir

Consul data directory, type: string, level: G, default value: "/data/consul".

ETCD

ETCD is used for HA PostgreSQL Leader election, alternative to Consul.

etcd_enabled

Enable etcd: bool, level: G, default value: true.

Setup etcd servers on nodes in dcs_servers, and write credential to all client nodes.

etcd_data_dir

ETCD data directory, type: string, level: G, default value: "/data/etcd".

28 - Config: Nodes

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty provides host provisioning and monitoring functions. The nodes.yml playbook can be executed to configure the node to the corresponding state and incorporate it into the Pigsty monitor system.

ID Name Section Type Level Comment
300 meta_node NODE_IDENTITY bool C mark this node as meta
301 nodename NODE_IDENTITY string I node instance identity
302 node_cluster NODE_IDENTITY string C node cluster identity
303 nodename_overwrite NODE_IDENTITY bool C overwrite hostname with nodename
304 nodename_exchange NODE_IDENTITY bool C exchange static hostname
310 node_etc_hosts_default NODE_DNS string[] C static DNS records
311 node_etc_hosts NODE_DNS string[] C/I extra static DNS records
312 node_dns_method NODE_DNS enum C how to setup dns service?
313 node_dns_servers NODE_DNS string[] C dynamic DNS servers
314 node_dns_options NODE_DNS string[] C /etc/resolv.conf options
320 node_repo_method NODE_PA enum C how to use yum repo (local)
321 node_repo_remove NODE_REPO bool C remove existing repo file?
322 node_repo_local_urls NODE_REPO url[] C local yum repo url list
330 node_packages_default NODE_PACKAGES string[] C pkgs to be installed on all node
331 node_packages NODE_PACKAGES string[] C extra pkgs to be installed
332 node_packages_meta NODE_PACKAGES string[] G meta node only packages
333 node_packages_meta_pip NODE_PACKAGES string G meta node pip3 packages
340 node_disable_firewall NODE_TUNE bool C disable firewall?
341 node_disable_selinux NODE_TUNE bool C disable selinux?
342 node_disable_numa NODE_TUNE bool C disable numa?
343 node_disable_swap NODE_TUNE bool C disable swap?
344 node_static_network NODE_TUNE bool C use static DNS config?
345 node_disk_prefetch NODE_TUNE bool C enable disk prefetch?
346 node_kernel_modules NODE_TUNE string[] C kernel modules to be installed
347 node_tune NODE_TUNE enum C node tune mode
348 node_sysctl_params NODE_TUNE dict C extra kernel parameters
350 node_data_dir NODE_ADMIN path C default data disk mountpoint
351 node_admin_enabled NODE_ADMIN bool G create admin user?
352 node_admin_uid NODE_ADMIN int G admin user UID
353 node_admin_username NODE_ADMIN string G admin user name
354 node_admin_ssh_exchange NODE_ADMIN bool C exchange admin ssh keys?
355 node_admin_pk_current NODE_ADMIN bool A pks to be added to admin
356 node_admin_pk_list NODE_ADMIN key[] C add current user’s pkey?
360 node_timezone NODE_TIME string C node timezone
361 node_ntp_enabled NODE_TIME bool C setup ntp on node?
362 node_ntp_service NODE_TIME enum C ntp mode: ntp or chrony?
363 node_ntp_servers NODE_TIME string[] C ntp server list
364 node_crontab_overwrite NODE_TIME bool C/I overwrite instead of append /etc/crontab
365 node_crontab NODE_TIME string[] C/I crontab list of node
370 docker_enabled DOCKER bool C docker enabled?
371 docker_cgroups_driver DOCKER string C docker cgroup driver
372 docker_registry_mirrors DOCKER string[] C docker registry mirror location
373 docker_image_cache DOCKER path C docker image cache tarball
380 node_exporter_enabled NODE_EXPORTER bool C node_exporter enabled?
381 node_exporter_port NODE_EXPORTER int C node_exporter listen port
382 node_exporter_options NODE_EXPORTER string C/I node_exporter extra cli args
390 promtail_enabled PROMTAIL bool C promtail enabled ?
391 promtail_clean PROMTAIL bool C/A remove promtail status file ?
392 promtail_port PROMTAIL int G promtail listen port
393 promtail_options PROMTAIL string C/I promtail cli args
394 promtail_positions PROMTAIL string C path to store promtail status file

NODE_IDENTITY

Each node has identity parameters that are configured through the parameters in <cluster>.hosts and <cluster>.vars.

Pigsty uses IP as a unique identifier for database nodes. This IP must be the IP that the database instance listens to and serves externally, but it is inappropriate to use a public IP. Users can also indirectly operate the management target node through an SSH tunnel or springboard machine transit. However, the primary IPv4 is still the core identity of the node when identifying the database node. This is very important. The IP is the inventory_hostname of the host in the inventory, which is reflected as the key in the <cluster>.hosts object.

In the Pigsty monitor system, nodes also have two crucial identity parameters: nodename and node_cluster. These will be used in the monitor system as the node’s instance identity (ins) and cluster identity (cls). Pigsty uses node-exclusive 1:1 deployment by default, so the identity params of the instances (pg_cluster and pg_instance) can be borrowed to the ins and cls tags of the nodes via the pg_hostname parameter.

nodename and node_cluster are not mandatory; when left blank or empty, nodename will use the node’s current hostname, while node_cluster will use the fixed default value: nodes.

Name Type Level Necessity Comment
inventory_hostname ip - Required Node IP
nodename string I Optional Node Name
node_cluster string C Optional Node cluster name

The following cluster config declares a three-node node cluster:

node-test:
  hosts:
    10.10.10.11: { nodename: node-test-1 }
    10.10.10.12: { nodename: node-test-2 }
    10.10.10.13: { nodename: node-test-3 }
  vars:
    node_cluster: node-test

meta_node

This node is a meta node, type: bool, level: C, default value: false.

Nodes under the meta grouping carry this flag in the inventory by default. Nodes with this flag will be additionally configured at node package installation with:

Install the RPM pkgs specified by node_packages_meta and install the Python pkgs set by node_packages_meta_pip.

nodename

Specifies the node name, type: string, level: I, the default value is null.

Null or empty string means nodename will be set to node’s current hostname.

no name is specified for the node, and the existing Hostname is used directly as the node name.

The node name nodename will be used as the name of the node instance (ins tag) in the Pigsty monitor system. In addition, if nodename_overwrite is true, the node name will also be used as the HOSTNAME.

Note: If the pg_hostname option is enabled, Pigsty will borrow the identity parameter of the one-by-one corresponding PG instance on the current node, such as pg-test-1, as the node name when initializing the node.

node_cluster

Node cluster name, type: string, level: C, default value: "nodes".

The default null value will directly use the fixed value nodes as the node cluster identity.

The node cluster name node_cluster will be used as the node cluster (cls) label in the Pigsty monitor system.

Note: If the pg_hostname option is enabled, Pigsty will borrow the identity parameter of the one-by-one corresponding PG cluster on the current node, such as pg-test, as the node cluster name when initializing the node.

nodename_overwrite

Override machine HOSTNAME with node name, type: bool, level: C, default value: true.

Defaults to true, a non-empty node name nodename will override the current hostname of the node.

No changes are made to the hostname if the nodename parameter is undefined, empty, or an empty string.

nodename_exchange

Exchange hostnames between playbook nodes, type: bool, level: C, default value: false.

When this parameter is enabled, node names are exchanged between the same group of nodes executing the nodes.yml playbook, written to /etc/hosts.

NODE_DNS

Pigsty configs static DNS records and dynamic DNS resolver for the nodes.

If you already have a DNS server, set node_dns_method to none to disable dynamic DNS setup.

node_etc_hosts

DNS records specific to the cluster instance level, type: string[], level: C/I, default value is an empty array [].

node_etc_hosts is an array. Each element is a string shaped like an ip domain_name, representing a DNS resolution record. Each of which is written to /etc/hosts when the machine node is initialized, suitable for cluster/instance specific records.

node_etc_hosts_default

Write to static DNS resolution of the machine, type: string[], level: C, default value:

node_etc_hosts_default:                 # static dns records in /etc/hosts
  - 10.10.10.10 meta pigsty c.pigsty g.pigsty l.pigsty p.pigsty a.pigsty cli.pigsty lab.pigsty api.pigsty

node_etc_hosts_default is an array. Each element is a string shaped like an ip domain_name, representing a DNS resolution record. Each of which is written to /etc/hosts when the machine node is initialized, suitable for global config of infra addresses.

Make sure to write a DNS record like 10.10.10.10 pigsty yum.pigsty to /etc/hosts to ensure that the local yum repo can be accessed using the domain name before the DNS Nameserver starts.

node_dns_method

Config DNS server, type: enum, level: C, default value: "add".

The default config of dynamic DNS servers for machine nodes has three modes:

  • add: Append the records in node_dns_servers to /etc/resolv.conf and keep the existing DNS servers. (default)
  • overwrite: Overwrite /etc/resolv.conf with the record in node_dns_servers
  • none: If a DNS server is provided in the production env, the DNS server config can be skipped.

node_dns_servers

Config dynamic DNS server list, type: string[], level: C, default value is 10.10.10.10.

Pigsty adds meta nodes as DNS Server by default, and DNSMASQ on the meta node responds to DNS requests in the env.

node_dns_servers: # dynamic nameserver in /etc/resolv.conf
  - 10.10.10.10

node_dns_options

If node_dns_method is configured as add or overwrite, the records in this config entry will be appended or overwritten to /etc/resolv.conf. Please see the Linux doc for /etc/resolv.conf for the exact format.

The default parsing options added by Pigsty:

- options single-request-reopen timeout:1 rotate
- domain service.consul

NODE_REPO

Pigsty configure yum repos & install packages from it.

node_repo_method

A node using Yum repo, type: enum, level: C, default value: "local".

The machine node Yum software repo is configured in three modes:

  • local: Use the local Yum repo on the meta node, the default behavior (recommended).
  • public: To install using internet sources, write the public repo in repo_upstream to /etc/yum.repos.d/.
  • none: No config and modification of local repos.

node_repo_remove

Remove nodes with existing Yum repos, type: bool, level: C, default value: true.

If enabled, Pigsty will move repo file in /etc/yum.repos.d to backup dir: /etc/yum.repos.d/backup

node_repo_local_urls

URL address of the local repo, type: url[], level: C, default value is local.

node_repo_method configured as local, the Repo file URLs listed here will be downloaded to /etc/yum.repos.d.

Here is an array of Repo File URLs that Pigsty will add by default to the machine’s source config for the local Yum repos on the meta node.

node_repo_local_urls:
  - http://yum.pigsty/pigsty.repo

NODE_PACKAGES

node_packages

List of extra installed software for the node, type: string[], level: C, default value:

There is a list of extra pkgs to install via yum, with an empty list by default.

Like node_packages_default, the former is usually configured globally, while node_packages makes exceptions for specific nodes.

node_packages_default

List of node installation software, type: string[], level: C, default value:

The package list is an array, but each element can contain multiple pkgs separated by commas. The list of pkgs installed by Pigsty by default is as follows:

node_packages_meta:                           # packages for meta nodes only
  - grafana,prometheus2,alertmanager,loki,nginx_exporter,blackbox_exporter,pushgateway,redis,postgresql14
  - nginx,ansible,pgbadger,python-psycopg2,dnsmasq,polysh,coreutils,diffutils

node_packages_meta

List of software required by the meta node, type: string[], level: G, default value:

node_packages_meta:                           # packages for meta nodes only
  - grafana,prometheus2,alertmanager,loki,nginx_exporter,blackbox_exporter,pushgateway,redis,postgresql14
  - nginx,ansible,pgbadger,python-psycopg2,dnsmasq,polysh,coreutils,diffutils

The pkgs listed in node_packages_meta will only be installed on the meta node, and infra software generally used on the meta node must be specified here.

node_packages_meta_pip

Package installed on the meta node via pip3, type: string, level: G, default value: "jupyterlab".

The package will be downloaded to {{ nginx_home }}/{{ repo_name }}/python dir and then installed uniformly.

Currently, jupyterlab will be installed by default, providing a complete Python runtime env.

NODE_TUNE

Configure some features, kernel modules, and tuning templates on the node.

node_disable_firewall

Turn off node firewall, type: bool, level: C, default value: true, please keep it off.

node_disable_selinux

Close node SELINUX, type: bool, level: C, default value: true, please keep it off.

node_disable_numa

Close the node NUMA, type: bool, level: C, default value: false.

Boolean flag, default is not off. Note that turning off NUMA requires a reboot of the machine before it can take effect!

If you don’t know how to set the affinity with a specific CPU core, it is recommended to turn off NUMA when using the database in a production env.

node_disable_swap

Turn off node SWAP, type: bool, level: C, default value: false.

Turning off SWAP is not recommended and can be done to improve performance if there is enough memory and the database is deployed exclusively.

SWAP should be disabled when your node is used for a Kubernetes deployment.

node_static_network

Use static DNS servers, Type: bool, Level: C, Default: true, Enabled by default.

Enabling static networking means that machine reboots will not overwrite your DNS Resolv config with NIC changes. It is recommended to allow for it.

node_disk_prefetch

Enable disk pre-reading, type: bool, level: C, default value: false, not enabled by default.

Instances deployed against HDDs optimize throughput and are recommended to be enabled when using HDDs.

node_kernel_modules

Enabled kernel module, type: string[], level: C, default value:

An array consisting of kernel module names declaring the kernel modules that need to be installed on the node. Pigsty will enable the following kernel modules by default:

node_kernel_modules: [ softdog, ip_vs, ip_vs_rr, ip_vs_rr, ip_vs_wrr, ip_vs_sh ]

node_tune

Node tuning mode, type: enum, level: C, default value: "tiny".

Prefabricated solutions for machine tuning, based on the tuned service. There are four pre-production models:

  • tiny： Micro Virtual Machine
  • oltp： Regular OLTP templates with optimized latency
  • olap ： Regular OLAP templates to optimize throughput
  • crit： Core financial business templates, optimizing the number of dirty pages

Usually, the database tuning template pg_conf should be paired with the machine tuning template. See Customize PGSQL Template for details.

node_sysctl_params

OS kernel parameter, type: dict, level: C, default value is an empty dictionary. Dictionary K-V structure, Key is kernel sysctl parameter name, Value is the parameter value.

NODE_ADMIN

Host Node Admin Users.

node_data_dir

Mountpoint of major data disk, level: C, default value: /data.

If specified, this path will be used as major data disk mountpoint.

And a dir will be created and throwing a warning if path not exists.

The data dir is owned by root with mode 0777.

node_admin_enabled

Create admin user, type: bool, level: G, default value: true.

To create an admin user on each node (password-free sudo and ssh), an admin user named dba (uid=88) will be created, which can access other nodes in the env and perform sudo from the meta node via SSH password-free.

node_admin_uid

Admin user UID, type: int, level: G, default value: 88, note UID namespace conflict.

node_admin_username

Admin username, type: string, level: G, default value: "dba".

node_admin_ssh_exchange

Exchange node admin SSH keys between instances, type: bool, level: C, default value: true.

When enabled, Pigsty will exchange SSH public keys between members during playbook execution, allowing admins node_admin_username to access each other from different nodes.

node_admin_pk_current

Whether to add the public key of the current node & user to the admin account, type: bool, level: A, default value: true.

When enabled, on the current node, the SSH public key (~/.ssh/id_rsa.pub) of the current user is copied to the authorized_keys of the target node admin user.

When deploying in a production env, be sure to pay attention to this parameter, which installs the default public key of the user currently executing the command to the admin user of all machines.

node_admin_pk_list

The list of public keys for login able admin, type: key[], level: C, default value is an empty array; the demo has the default public key for vagrant users.

Each element of the array is a string containing the key written to the admin user ~/.ssh/authorized_keys, and the user with the corresponding private key can log in as an admin user.

When deploying in production envs, be sure to note this parameter and add only trusted keys to this list.

NODE_TIME

The node time zone is synchronized with time.

If the node is already configured with an NTP server, you can configure node_ntp_enabled to false to skip the setting of the NTP service.

node_timezone

Node time zone, type: string, level: C, default value: "Asia/Hong_Kong".

The default time zone used in the demo is "Asia/Hong_Kong" please adjust it according to your actual situation. (Please don’t use Asia/Shanghai time zone, the abbreviation CST will cause a series of log time zone parsing problems)

Select false, or leave it blank, and Pigsty will not modify the time zone config of this node.

node_ntp_enabled

Is the NTP service configured? , type: bool, level: C, default value: true.

Value is true: Pigsty will override the node’s /etc/ntp.conf or /etc/chrony.conf by filling in the NTP server specified by node_ntp_servers.

If the server node is already configured with an NTP server, it is recommended to turn it off and use the original NTP server.

node_ntp_service

NTP service type: ntp or chrony, type: enum, level: C, default value: "ntp".

Specify the type of NTP service used by the system; by default, ntp is used as the time service:

  • ntp： Traditional NTP Service
  • chrony： Time services used by CentOS 7/8 by default

It only takes effect if node_ntp_enabled is true.

node_ntp_servers

List of NTP servers, type: string[], level: C, default value:

- pool cn.pool.ntp.org iburst
- pool pool.ntp.org iburst
- pool time.pool.aliyun.com iburst
- server 10.10.10.10 iburst

It only takes effect if node_ntp_enabled is true.

node_crontab_overwrite

Overwrite instead of append /etc/crontab, type: bool, level: C/I, default value: true

if true, records in node_crontab will render to/etc/crontab instead of appending to it.

node_crontab

Crontab of this node, type: string[], level: C/I, default value: []

Each element of the array is a string in /etc/crontab.

DOCKER

Pigsty install docker on all meta nodes by default, disabled on common nodes by default.

docker_enabled

Enable docker on current node? type: bool, level: C, default value: false. default true on meta nodes.

docker_cgroups_driver

CGroup drivers for docker, type: string, level: C, default value: systemd.

docker_registry_mirrors

Docker registry mirror list, type: string[], level：C, default value: [].

docker_image_cache

Local image cache, type: path, level: C, default value: "/tmp/docker.tgz".

The local image cache will be loaded into docker when the target path exists.

NODE_EXPORTER

NodeExporter is used to collect monitor metrics data from the host.

node_exporter_enabled

Enable node indicator collector, type: bool, level: C, default value: true.

node_exporter_port

NodeExposure Port, type: int, level: C, default value: 9100.

node_exporter_options

Node metrics collection option, type: string, level: C/I, default value: "--no-collector.softnet --no-collector.nvme --collector.ntp --collector.tcpstat --collector.processes"

Pigsty enables ntp, tcpstat, processes three extra metrics, collectors, by default, and disables softnet, nvme two default metrics collectors.

PROMTAIL

Host log collection component, used with Loki infrastructure config.

promtail_enabled

Enable Protail log collection service at the current node, type: bool, level: C, default value: true.

When promtail is enabled, Pigsty will generate a config file for Promtail, as defined in the inventory, to grab the following logs and send them to the Loki instance specified by loki_endpoint.

  • INFRA： Infra logs, collected only on meta nodes.

    • nginx-access: /var/log/nginx/access.log
    • nginx-error: /var/log/nginx/error.log
    • grafana: /var/log/grafana/grafana.log

  • NODES： Host node logs, collected on all nodes.

    • syslog: /var/log/messages
    • dmesg: /var/log/dmesg
    • cron: /var/log/cron

  • PGSQL： PostgreSQL logs, collected when a node is defined with pg_cluster.

    • postgres: /pg/data/log/*.csv
    • patroni: /pg/log/patroni.log
    • pgbouncer: /var/log/pgbouncer/pgbouncer.log

  • REDIS： Redis logs, collected when a node is defined with redis_cluster.

    • redis: /var/log/redis/*.log

promtail_clean

Remove existing state information when installing protail, type: bool, level: C/A, default value: false.

The default is not to clean up; when you choose to clean up, Pigsty will remove the existing state file promtail_positions when deploying Promtail, which means that Promtail will recollect all logs on the current node and send them to Loki.

promtail_port

The default port used by promtail, type: int, level: G, default value: 9080.

promtail_options

Promtail CLI param, type: string, level: C/I, default value: "-config.file=/etc/promtail.yml -config.expand-env=true".

Extra CLI params passed in when running the protail database, default value: '-config.file=/etc/promtail.yml -config.expand-env=true'.

There are already params for specifying the config file path and expanding the environment variables in the config file, which are not recommended to be modified.

promtail_positions

Path to promtail status file, type: string, level: C, default value："/var/log/positions.yaml"

Promtail records the consumption offsets of all logs, which are periodically written to the file specified by promtail_positions.

29 - Config: PGSQL

Historical documentation restored from the Pigsty v1.5.1 tag.

Use the PGSQL Playbook, and deploy the PGSQL cluster to adjust the cluster state to the state described in the PGSQL config.

Pigsty has 100+ config parameters for describing a PostgreSQL cluster. However, users usually only need to care about a few parameters in identity params and business objects: the former expresses the database cluster “Who is it? Where is it?” and the latter represents the database “What does it look like? What’s in it?”.

The params on the PostgreSQL are divided into seven main sections：

  • PG_IDENTITY: Defining the identity of a PostgreSQL cluster.
  • PG_BUSINESS: Customized cluster templates: users, databases, services, privilege rules.
  • PG_INSTALL: Install PostgreSQL pkgs, extension plugins, and prepare dir and tool scripts.
  • PG_BOOTSTRP: Generate config template, pull up PostgreSQL cluster, build M-S replication, and enable connection pooling.
  • PG_PROVISION: PGSQL cluster template provisioning, creating users and databases, configuring privileges role HBA, mode and extensions.
  • PG_EXPORTER: PGSQL-exporter, database, and connection pool config monitoring component.
  • PG_SERVICE: Expose the PostgreSQL service, install the LB HAProxy, enable VIP, and configure DNS.
ID Name Section Type Level Comment
500 pg_cluster PG_IDENTITY string C PG Cluster Name
501 pg_shard PG_IDENTITY string C PG Shard Name (Reserve)
502 pg_sindex PG_IDENTITY int C PG Shard Index (Reserve)
503 gp_role PG_IDENTITY enum C gp role of this PG cluster
504 pg_role PG_IDENTITY enum I PG Instance Role
505 pg_seq PG_IDENTITY int I PG Instance Sequence
506 pg_instances PG_IDENTITY {port:ins} I PG instance on this node
507 pg_upstream PG_IDENTITY string I PG upstream IP
508 pg_offline_query PG_IDENTITY bool I allow offline query?
509 pg_backup PG_IDENTITY bool I make base backup on this ins?
510 pg_weight PG_IDENTITY int I relative weight in LB
511 pg_hostname PG_IDENTITY bool C/I set PG ins name as hostname
512 pg_preflight_skip PG_IDENTITY bool C/A skip preflight param validation
520 pg_users PG_BUSINESS user[] C business users definition
521 pg_databases PG_BUSINESS database[] C business databases definition
522 pg_services_extra PG_BUSINESS service[] C ad hoc service definition
523 pg_hba_rules_extra PG_BUSINESS rule[] C ad hoc HBA rules
524 pgbouncer_hba_rules_extra PG_BUSINESS rule[] C ad hoc pgbouncer HBA rules
525 pg_admin_username PG_BUSINESS string G admin user’s name
526 pg_admin_password PG_BUSINESS string G admin user’s password
527 pg_replication_username PG_BUSINESS string G replication user’s name
528 pg_replication_password PG_BUSINESS string G replication user’s password
529 pg_monitor_username PG_BUSINESS string G monitor user’s name
530 pg_monitor_password PG_BUSINESS string G monitor user’s password
540 pg_dbsu PG_INSTALL string C os dbsu for postgres
541 pg_dbsu_uid PG_INSTALL int C dbsu UID
542 pg_dbsu_sudo PG_INSTALL enum C sudo priv mode for dbsu
543 pg_dbsu_home PG_INSTALL path C home dir for dbsu
544 pg_dbsu_ssh_exchange PG_INSTALL bool C exchange dbsu ssh keys?
545 pg_version PG_INSTALL int C major PG version to be installed
546 pgdg_repo PG_INSTALL bool C add official PGDG repo?
547 pg_add_repo PG_INSTALL bool C add extra upstream PG repo?
548 pg_bin_dir PG_INSTALL path C PG binary dir
549 pg_packages PG_INSTALL string[] C PG packages to be installed
550 pg_extensions PG_INSTALL string[] C PG extension pkgs to be installed
560 pg_safeguard PG_BOOTSTRAP bool C/A disable pg instance purge
561 pg_clean PG_BOOTSTRAP bool C/A purge existing pgsql during init
562 pg_data PG_BOOTSTRAP path C pg data dir
563 pg_fs_main PG_BOOTSTRAP path C pg main data disk mountpoint
564 pg_fs_bkup PG_BOOTSTRAP path C pg backup disk mountpoint
565 pg_dummy_filesize PG_BOOTSTRAP size C /pg/dummy file size
566 pg_listen PG_BOOTSTRAP ip C pg listen IP
567 pg_port PG_BOOTSTRAP int C pg listen port
568 pg_localhost PG_BOOTSTRAP ip path C
580 patroni_enabled PG_BOOTSTRAP bool C Is patroni & postgres enabled?
581 patroni_mode PG_BOOTSTRAP enum C patroni working mode
582 pg_dcs_type PG_BOOTSTRAP enum G dcs to be used consul/etcd
583 pg_namespace PG_BOOTSTRAP path C namespace for patroni
584 patroni_port PG_BOOTSTRAP int C patroni listen port (8080)
585 patroni_watchdog_mode PG_BOOTSTRAP enum C patroni watchdog policy
586 pg_conf PG_BOOTSTRAP string C patroni template
587 pg_libs PG_BOOTSTRAP string C default preload shared database
588 pg_delay PG_BOOTSTRAP interval I apply delay for standby leader
589 pg_checksum PG_BOOTSTRAP bool C enable data checksum
590 pg_encoding PG_BOOTSTRAP enum C character encoding
591 pg_locale PG_BOOTSTRAP enum C locale
592 pg_lc_collate PG_BOOTSTRAP enum C collate rule of locale
593 pg_lc_ctype PG_BOOTSTRAP enum C ctype of locale
594 pgbouncer_enabled PG_BOOTSTRAP bool C is pgbouncer enabled
595 pgbouncer_port PG_BOOTSTRAP int C pgbouncer listen port
596 pgbouncer_poolmode PG_BOOTSTRAP enum C pgbouncer pooling mode
597 pgbouncer_max_db_conn PG_BOOTSTRAP int C max connection per database
600 pg_provision PG_PROVISION bool C provision template to pgsql?
601 pg_init PG_PROVISION string C path to postgres init script
602 pg_default_roles PG_PROVISION role[] G/C list or global default roles/users
603 pg_default_privileges PG_PROVISION string[] G/C list of default privileges
604 pg_default_schemas PG_PROVISION string[] G/C list of default modes
605 pg_default_extensions PG_PROVISION extension[] G/C list of default extensions
606 pg_reload PG_PROVISION bool A reload config?
607 pg_hba_rules PG_PROVISION rule[] G/C global HBA rules
608 pgbouncer_hba_rules PG_PROVISION rule[] G/C global pgbouncer HBA rules
620 pg_exporter_config PG_EXPORTER string C pg_exporter config path
621 pg_exporter_enabled PG_EXPORTER bool C pg_exporter enabled ?
622 pg_exporter_port PG_EXPORTER int C pg_exporter listen address
623 pg_exporter_params PG_EXPORTER string C/I extra params for pg_exporter url
624 pg_exporter_url PG_EXPORTER string C/I monitor target pgurl (overwrite)
625 pg_exporter_auto_discovery PG_EXPORTER bool C/I enable auto-database-discovery?
626 pg_exporter_exclude_database PG_EXPORTER string C/I excluded list of databases
627 pg_exporter_include_database PG_EXPORTER string C/I included list of databases
628 pg_exporter_options PG_EXPORTER string C/I cli args for pg_exporter
629 pgbouncer_exporter_enabled PG_EXPORTER bool C pgbouncer_exporter enabled ?
630 pgbouncer_exporter_port PG_EXPORTER int C pgbouncer_exporter listen addr?
631 pgbouncer_exporter_url PG_EXPORTER string C/I target pgbouncer url (overwrite)
632 pgbouncer_exporter_options PG_EXPORTER string C/I cli args for pgbouncer exporter
640 pg_services PG_SERVICE service[] G/C global service definition
641 haproxy_enabled PG_SERVICE bool C/I haproxy enabled ?
642 haproxy_reload PG_SERVICE bool A haproxy reload instead of reset
643 haproxy_auth_enabled PG_SERVICE bool G/C enable auth for haproxy admin ?
644 haproxy_admin_username PG_SERVICE string G haproxy admin user name
645 haproxy_admin_password PG_SERVICE string G haproxy admin password
646 haproxy_exporter_port PG_SERVICE int C haproxy exporter listen port
647 haproxy_client_timeout PG_SERVICE interval C haproxy client timeout
648 haproxy_server_timeout PG_SERVICE interval C haproxy server timeout
649 vip_mode PG_SERVICE enum C vip working mode
650 vip_reload PG_SERVICE bool A reload vip configuration
651 vip_address PG_SERVICE string C vip address used by cluster
652 vip_cidrmask PG_SERVICE int C vip network CIDR length
653 vip_interface PG_SERVICE string C vip network interface name
654 dns_mode PG_SERVICE enum C cluster DNS mode
655 dns_selector PG_SERVICE string C cluster DNS ins selector

PG_IDENTITY

pg_cluster, pg_role, pg_seq belong to identity params .

In addition to the IP, these three parameters are the minimum set of parameters necessary to define a new set of clusters. A typical example is shown below:

pg-test:
  hosts:
    10.10.10.11: {pg_seq: 1, pg_role: replica}
    10.10.10.12: {pg_seq: 2, pg_role: primary}
    10.10.10.13: {pg_seq: 3, pg_role: replica}
  vars:
    pg_cluster: pg-test

All other params can be inherited from the global config or the default config, but the identity params must be explicitly specified and manually assigned. The current PGSQL identity params are as follows:

Name Type Level Description
pg_cluster string C PG database cluster name
pg_seq number I PG database ins serial number
pg_role enum I PG database ins role
pg_shard string C PG database slice set cluster name (placeholder)
pg_sindex number C PG database slice set cluster number (placeholder)
  • pg_cluster: It identifies the name of the cluster, which is configured at the cluster level.
  • pg_role: Configured at the instance level, identifies the role of the ins. Only the primary role will be handled specially. If not filled in, the default is the replica role and the special delayed and offline roles.
  • pg_seq: Used to identify the ins within the cluster, usually with an integer number incremented from 0 or 1, which is not changed once it is assigned.
  • {{ pg_cluster }}-{{ pg_seq }} is used to uniquely identify the ins, i.e. pg_instance.
  • {{ pg_cluster }}-{{ pg_role }} is used to identify the services within the cluster, i.e. pg_service.
  • pg_shard and pg_sindex are used for horizontally sharding clusters, reserved for Citus and Greenplum multicluster management.

pg_cluster

PG cluster name, type: string, level: cluster, no default. A mandatory parameter must be provided by the user.

The cluster name will be used as the namespace for the resources within the cluster. The naming needs to follow a specific naming pattern: [a-z][a-z0-9-]* to be compatible with the requirements of different constraints on the identity.

pg_shard

Shard to which the PG cluster belongs (reserved), type: string, level: cluster, No default.

Only sharding clusters require this parameter to be set. When multiple clusters serve the same business in a horizontally sharded fashion, Pigsty refers to this group of clusters as a Sharding Cluster.

pg_shard is the name of the shard set cluster to which the cluster belongs. A shard set cluster can be specified with any name, but Pigsty recommends a meaningful naming pattern.

For example, a cluster participating in a sharded cluster can use the shard cluster name pg_shard + shard + the cluster’s shard number pg_sindex to form the cluster name：

shard:  test
pg-testshard1
pg-testshard2
pg-testshard3
pg-testshard4

pg_sindex

PG cluster’s slice number (reserved), type: int, level: C, no default.

The sharded cluster’s slice number, used in conjunction with pg_shard is usually assigned sequentially starting from 0 or 1. Only sharded clusters require this param to be set.

gp_role

Current role of PG cluster in GP, type: enum, level: C, default value：

Greenplum/MatrixDB-specific to specify the role this PG cluster plays in a GP deployment. The optional values are ：

  • master: Facilitator Nodes
  • segment: Data Nodes

Identity parameter, cluster level parameter, and mandatory parameter when deploying GPSQL.

pg_role

PG instance role, type: enum, level: I, no default, mandatory parameter, must be provided by the user.

Roles for PG ins, default roles include primary, replica, and offline.

  • primary: Primary, there must be one and only one member of the cluster as primary.
  • replica: Replica for carrying online read-only traffic.
  • offline: Offline replica for taking on offline read-only traffic, such as statistical analysis/ETL/personal queries, etc.

Identity params, required params, and instance-level params.

pg_seq

PG ins serial number, type: int, level: I, no default value, mandatory parameter, must be provided by the user.

A serial number of the database ins, unique within the cluster, is used to distinguish and identify different instances within the cluster, assigned starting from 0 or 1.

pg_instances

All PG instances on the current node, type: {port:ins}, level: I, default value：

This parameter can be used to describe when the node is deployed by more than one PG ins, such as Greenplum’s Segments, or when using monly mode to supervise existing ins. pg_instances is an array of objects with keys as ins ports and values as a dictionary whose contents can be parameters of any PGSQL board, see MatrixDB deploy for details.

pg_upstream

The replicated upstream node of the instance, type: string, level: I, the default value is null.

Ins-level config entry with IP or hostname to indicate the upstream node for stream replication.

  • When configuring this parameter for a replica, the IP filled in must be another node within the cluster. Instances will be stream replicated from that node, and this option can be used to build cascaded replication.

  • When this parameter is configured for the primary of the cluster, it means that the entire cluster will run as a Standby Cluster, receiving changes from upstream nodes. The primary in the cluster will play the role of standby leader.

Using this parameter flexibly, you can build a standby cluster, complete the splitting of the sharded cluster, and realize the delayed cluster.

pg_offline_query

Allow offline queries, type: bool, level: I, default value: false.

When set to true, the user group dbrole_offline can connect to the ins and perform offline queries, regardless of the role of the current ins.

More practical for cases with a small number of ins (one primary & one replica), the user can mark the only replica as pg_offline_query = true, thus accepting ETL, slow queries with interactive access.

pg_backup

Store cold standbys on the ins, type: bool, level: I, default value: false.

Not implemented, the tag bit is reserved and the ins node with this tag is used to store the base cold standby.

pg_weight

The relative weight of the ins in load balancing, type: int, level: I, default value: 100.

When adjusting the relative weight of an instance in service, this parameter can be modified at the instance level and applied to take effect as described in SOP: Cluster Traffic Adjustment.

pg_hostname

Set PG ins name to HOSTNAME, type: bool, level: C/I, default value: true.

Use the PG ins name and cluster name as the node’s name and cluster name when initializing the nodean , enabled by default in the bundled v1.5.1 inventory.

When using the node: PG 1:1 exclusive deploy mode, you can assign the identity of the PG ins to the node, making the node consistent with the PG’s monitor identity.

pg_preflight_skip

Skip preflight param validation, type: bool, level: C/A, default value: false.

If not initializing a new cluster, the task of Patroni and Postgres initialization can be completely skipped with this parameter.

PG_BUSINESS

Users need to focus on this part of the parameters to declare their required database objects on behalf of the business.

Customized cluster templates: users, databases, services, privilege patterns.

Special DB users, it is recommended to change these user passwords in the production env.

pg_users

Business user definition, type: user[], level: C, default value is an empty array.

Used to define business users at the cluster level, each object in the array defines a user or role, a complete user definition is as follows:

pg_users:                           # define business users/roles on this cluster, array of user definition
  # define admin user for meta database (This user are used for pigsty app deployment by default)
  - name: dbuser_meta               # required, `name` is the only mandatory field of a user definition
    password: md5d3d10d8cad606308bdb180148bf663e1  # md5 salted password of 'DBUser.Meta'
    # optional, plain text and md5 password are both acceptable (prefixed with `md5`)
    login: true                     # optional, can login, true by default  (new biz ROLE should be false)
    superuser: false                # optional, is superuser? false by default
    createdb: false                 # optional, can create database? false by default
    createrole: false               # optional, can create role? false by default
    inherit: true                   # optional, can this role use inherited privileges? true by default
    replication: false              # optional, can this role do replication? false by default
    bypassrls: false                # optional, can this role bypass row level security? false by default
    pgbouncer: true                 # optional, add this user to pgbouncer user-list? false by default (production user should be true explicitly)
    connlimit: -1                   # optional, user connection limit, default -1 disable limit
    expire_in: 3650                 # optional, now + n days when this role is expired (OVERWRITE expire_at)
    expire_at: '2030-12-31'         # optional, YYYY-MM-DD 'timestamp' when this role is expired  (OVERWRITTEN by expire_in)
    comment: pigsty admin user      # optional, comment string for this user/role
    roles: [dbrole_admin]           # optional, belonged roles. default roles are: dbrole_{admin,readonly,readwrite,offline}
    parameters:                     # optional, role level parameters with `ALTER ROLE SET`
      log_min_duration_statements: 1000
    search_path: public         # key value config parameters according to postgresql documentation (e.g: use pigsty as default search_path)
  - {name: dbuser_view , password: DBUser.Viewer  ,pgbouncer: true ,roles: [dbrole_readonly], comment: read-only viewer for meta database}

  # define additional business users for prometheus & grafana (optional)
  - {name: dbuser_grafana    , password: DBUser.Grafana    ,pgbouncer: true ,roles: [dbrole_admin], comment: admin user for grafana database }
  - {name: dbuser_prometheus , password: DBUser.Prometheus ,pgbouncer: true ,roles: [dbrole_admin], comment: admin user for prometheus database }
  • Each user or role must specify a name and the rest of the fields are optional, a name must be unique in this list.
  • password is optional, if left blank then no password is set, you can use the MD5 ciphertext password.
  • login, superuser, createdb, createrole, inherit, replication and bypassrls are all boolean types used to set user attributes. If not set, the system defaults are used.
  • Users are created by CREATE USER, so they have the login attribute by default. If the role is created, you need to specify login: false.
  • expire_at and expire_in are used to control the user expiration time. expire_at uses a date timestamp in the shape of YYYY-mm-DD. expire_in uses the number of days to expire from now, and overrides the expire_at option if expire_in exists.
  • New users are not added to the Pgbouncer user list by default, and pgbouncer: true must be explicitly defined for the user to be added to the Pgbouncer user list.
  • Users/roles are created sequentially, and users defined later can belong to the roles defined earlier.
  • Users can add default privilegs groups for business users via the roles field:
    • dbrole_readonly: Default production read-only user with global read-only privileges. (Read-only production access)
    • dbrole_offline: Default offline read-only user with read-only access on a specific ins. (offline query, personal account, ETL)
    • dbrole_readwrite: Default production read/write user with global CRUD privileges. (Regular production use)
    • dbrole_admin: Default production management user with the privilege to execute DDL changes. (Admin User)

Configure pgbouncer: true for the production account to allow it to access through the connection pool; regular users should not access the database through the connection pool.

pg_databases

Business database definition, type: database[], level: C, default value is an empty array.

Used to define business users at the cluster level, each object in the array defines a business database, a complete database definition as follows:

pg_databases:                       # define business databases on this cluster, array of database definition
  # define the default `meta` database
  - name: meta                      # required, `name` is the only mandatory field of a database definition
    baseline: cmdb.sql              # optional, database sql baseline path, (relative path among ansible search path, e.g files/)
    owner: postgres                 # optional, database owner, postgres by default
    template: template1             # optional, which template to use, template1 by default
    encoding: UTF8                  # optional, database encoding, UTF8 by default. (MUST same as template database)
    locale: C                       # optional, database locale, C by default.  (MUST same as template database)
    lc_collate: C                   # optional, database collate, C by default. (MUST same as template database)
    lc_ctype: C                     # optional, database ctype, C by default.   (MUST same as template database)
    tablespace: pg_default          # optional, default tablespace, 'pg_default' by default.
    allowconn: true                 # optional, allow connection, true by default. false will disable connect at all
    revokeconn: false               # optional, revoke public connection privilege. false by default. (leave connect with grant option to owner)
    pgbouncer: true                 # optional, add this database to pgbouncer database list? true by default
    comment: pigsty meta database   # optional, comment string for this database
    connlimit: -1                   # optional, database connection limit, default -1 disable limit
    schemas: [pigsty]               # optional, additional schemas to be created, array of schema names
    extensions:                     # optional, additional extensions to be installed: array of schema definition `{name,schema}`
      - {name: adminpack, schema: pg_catalog}    # install adminpack to pg_catalog and install postgis to public
      - {name: postgis, schema: public}          # if schema is omitted, extension will be installed according to search_path.

In each DB definition, the DB name is mandatory and the rest are optional.

  • name: Database name, Must.
  • owner: Database owner, default is postgres
  • template: The template used for database creation, default is template1.
  • encoding: The default character encoding of the database, which is UTF8 by default, is consistent with the ins by default. It is recommended not to configure and modify it.
  • locale: The default localization rule for the database, which defaults to C, is recommended not to be configured to be consistent with the instance.
  • lc_collate: The default localized string sorting rule for the database, which is set the same as the instance by default, should not be modified and must be consistent with the DB template. It is strongly recommended not to configure, or configure to C.
  • lc_ctype: The default LOCALE of the database, by default, is the same as the ins setting, do not modify or set it, it must be consistent with the DB template. Configure to C or en_US.UTF8.
  • allowconn: Allow database connection, default is true, not recommended to change.
  • revokeconn: Reclaim privilege to connect to the database. The default is false. To be true, the PUBLIC CONNECT privilege on the database will be reclaimed. Only the default user (dbsu|monitor|admin|replicator|owner) can connect. In addition, the admin|owner will have GRANT OPTION, which can give other users connection privileges.
  • tablespace: The tablespace associated with the database, the default is pg_default.
  • connlimit: Database connection limit, default is -1, i.e. no limit.
  • extensions: An array of objects, each of which defines an extension in the database, and its installed mode.
  • parameters: K-V objects, each K-V defines a parameter that needs to be modified against the database via ALTER DATABASE.
  • pgbouncer: Boolean option to join this database to Pgbouncer or not. All databases are joined to Pgbouncer unless pgbouncer: false is explicitly specified.
  • comment: Database note information.

pg_services_extra

Cluster Proprietary Service Definition, Type: service[], Level: C, Default:

Used to define additional services at the cluster level, each object in the array defines a service, a complete service definition is as follows:

- name: default           # service's actual name is {{ pg_cluster }}-{{ service.name }}
  src_ip: "*"             # service bind ip address, * for all, vip for cluster virtual ip address
  src_port: 5436          # bind port, mandatory
  dst_port: postgres      # target port: postgres|pgbouncer|port_number , pgbouncer(6432) by default
  check_method: http      # health check method: only http is available for now
  check_port: patroni     # health check port:  patroni|pg_exporter|port_number , patroni by default
  check_url: /primary     # health check url path, / as default
  check_code: 200         # health check http code, 200 as default
  selector: "[]"          # instance selector
  haproxy:                # haproxy specific fields
    maxconn: 3000         # default front-end connection
    balance: roundrobin   # load balance algorithm (roundrobin by default)
    default_server_options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

Each cluster can define multiple services, each containing any number of cluster members. Services are distinguished by port, name, and src_port are mandatory and must be unique within the array.

MUST OPTION

  • Name（service.name）：

    The full name of the service is prefixed by the cluster name and suffixed by service.name, connected by -. For example, the service with name=primary in the pg-test cluster has the full-service name pg-test-primary.

  • Port（service.port）：

    In Pigsty, services are exposed to the public by default in the form of NodePort, so exposing the port is mandatory. However, if you use an external LB service access scheme, you can also differentiate the services in other ways.

  • Selector（service.selector）：

    The selector specifies the ins members of the service, in the form of JMESPath, filtering variables from all cluster ins members. The default [] selector will pick all cluster members.

Optional

  • Backup Selector（service.selector）：

    The optional backup selector service.selector_backup selects or marks the list of ins used for service backup, i.e. the backup ins take over the service only when all other members of the cluster fail. For example, the primary ins can be added to the replica service’s alternative set, so that the primary can still carry the cluster’s read-only traffic when all replicas fail.

  • Source IP（service.src_ip） ：

    Indicates the IP used externally by the service. The default is *, which is all IPs on the local. Using vip will use the vip_address variable to take the value, or you can fill in the specific IP supported by the NIC.

  • Host port（service.dst_port）：

    Which port on the target ins will the service’s traffic be directed to? postgres will point to the port the database is listening on, pgbouncer will point to the port the connection pool is listening on, or you can fill in a fixed port.

  • Health Check method（service.check_method）:

    How does the service check the health status of the instance? Currently, only HTTP is supported.

  • Health Check Port（service.check_port）:

    Which port of the service check-ins gets the health status of the ins? patroni will get it from Patroni (default 8008), pg_exporter will get it from PG Exporter (default 9630), or you can fill in a custom port.

  • Health Check Path（service.check_url）:

    The service performs HTTP checks using the URL PATH. / is used as a health check by default, and PG Exporter and Patroni provide a variety of health checks that can be used to differentiate between primary & replica traffic. For example, /primary will only return success for the primary, and /replica will only return success for the replica. /read-only will return success for any instance that supports read-only (including the primary).

  • Health Check Code（service.check_code）:

    The code expected by the HTTP health check, default is 200.

  • Haproxy Specific Placement（service.haproxy） ：

    Proprietary config entries for service provisioning software (HAproxy).

    • <service>.haproxy

    These parameters are now defined in service, using service.haproxy to override the parameter config of the ins.

    • maxconn

    HAProxy maximum number of front and back-end connections, default is 3000.

    • balance

    In the algorithm used by haproxy LB, the optional policy is roundrobin, and leastconn, the default is roundrobin.

    • default_server_options

    Default options for Haproxy backend server ins:

    'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

pg_hba_rules_extra

Cluster/ins specific HBA rule, Type: rule[], Level: C, Default:

Set the client IP black and white list rules for the database. An array of objects, each of which represents a rule, each of which consists of three parts:

  • title: Rule headings, which are converted to comments in the HBA file
  • role: Apply for roles, common means apply to all instances, other values (e.g. replica, offline) will only be installed to matching roles. For example, role='replica' means that this rule will only be applied to instances with pg_role == 'replica'.
  • rules: Array of strings, each record represents a rule that will eventually be written to pg_hba.conf.

As a special case, the HBA rule for role == 'offline' is additionally installed on instance of pg_offline_query == true.

pg_hba_rules is similar, but is typically used for global uniform HBA rule settings, and pg_hba_rules_extra will append to pg_hba.conf in the same way.

If you need to completely overwrite the cluster’s HBA rules and do not want to inherit the global HBA config, you should configure pg_hba_rules at the cluster level and override the global config.

pgbouncer_hba_rules_extra

Pgbounce HBA rule, type: rule[], level: C, default value is an empty array.

Similar to pg_hba_rules_extra for extra config of Pgbouncer’s HBA rules at the cluster level.

pg_admin_username

PG admin user, type: string, level: G, default value: "dbuser_dba".

The DB username is used to perform PG management tasks (DDL changes), with superuser privileges by default.

pg_admin_password

PG admin user password, type: string, level: G, default value: "DBUser.DBA".

The database user password used to perform PG management tasks (DDL changes) must be in plaintext. The default is DBUser.DBA and highly recommended changes!

It is highly recommended to change this parameter when deploying in production envs!

pg_replication_username

PG replication user’s name, type: string, level: G, default value: "replicator".

For performing PostgreSQL stream replication, it is recommended to keep global consistency.

pg_replication_password

PG’s Replication User Password, type: string, level: G, default value: "DBUser.Replicator".

The password of the database user used to perform PostgreSQL stream replication must be in plaintext. The default is DBUser.Replicator.

It is highly recommended to change this parameter when deploying in production envs!

pg_monitor_username

PG monitor user, type: string, level: G, default value: "dbuser_monitor".

The database user name is used to perform PostgreSQL and Pgbouncer monitoring tasks.

pg_monitor_password

PG monitor user password, type: string, level: G, default value: "DBUser.Monitor".

The password of the database user used to perform PostgreSQL and Pgbouncer monitoring tasks, must be in plaintext.

It is highly recommended to change this parameter when deploying in production envs!

PG_INSTALL

PG Install is responsible for completing the installation of all PostgreSQL dependencies on a machine with the base software. The user can configure the name, ID, privileges, and access of the dbsu, configure the sources used for the installation, configure the installation address, the version to be installed, and the required pkgs and extensions plugins.

Such parameters only need to be modified when upgrading a major version of the database as a whole. Users can specify the software version to be installed via pg_version and override it at the cluster level to install different database versions for different clusters.

pg_dbsu

PG OS dbsu, type: string, level: C, default value: "postgres", not recommended to modify.

When installing Greenplum / MatrixDB, modify this parameter to the corresponding recommended value: gpadmin|mxadmin.

pg_dbsu_uid

dbsu UID, type: int, level: C, default value: 26.

UID of the dbsu is used by the database by default. The default value is 26, consistent with the official RPM pkg-config of PG under CentOS, no modification is recommended.

pg_dbsu_sudo

Sudo privilege for dbsu, type: enum, level: C, default value: "limit".

  • none: No Sudo privilege
  • limit: Limited sudo privilege to execute systemctl commands for database-related components, default.
  • all: Full sudo privilege, password required.
  • nopass: Full sudo privileges without a password (not recommended).

The database superuser pg_dbsu has restricted sudo privilege by default: limit.

pg_dbsu_home

Home dir of dbsu pg_dbsu, type: path, level: C, default value: "/var/lib/pgsql".

pg_dbsu_ssh_exchange

Exchange the SSH key of dbsu pg_dbsu between executing machines. Type: bool, Level: C, Default: true.

pg_version

Installed major PG version, type: int, level: C, default value: 14.

The current instance’s installed a major PG version. Default is 14, supported as low as 10.

Note that PostgreSQL physical stream replication cannot span major versions, please configure this variable at the global/cluster level to ensure that all ins within the entire cluster have the same major version number.

pgdg_repo

Add the official PG repo? , type: bool, level: C, default value: false.

Use this option to download and install PostgreSQL-related pkgs directly from official Internet repos without local repos.

pg_add_repo

Add PG-related upstream repos? , type: bool, level: C, default value: false

If used, the official PGDG repo will be added before installing PostgreSQL.

pg_bin_dir

PG binary dir, type: path, level: C, default value: "/usr/pgsql/bin".

The default value is a softlink created manually during the installation process, pointing to the specific Postgres version dir installed.

For example /usr/pgsql -> /usr/pgsql-14. For more details, please see FHS.

pg_packages

List of installed PG pkgs, type: string[], level: C, default value:

- postgresql${pg_version}*
- postgis32_${pg_version}*
- citus_${pg_version}*
- timescaledb-2-postgresql-${pg_version}
- pgbouncer pg_exporter pgbadger pg_activity node_exporter consul haproxy vip-manager
- patroni patroni-consul patroni-etcd python3 python3-psycopg2 python36-requests python3-etcd
- python3-consul python36-urllib3 python36-idna python36-pyOpenSSL python36-cryptography

${pg_version} in the package will be replaced with the actual installed PostgreSQL version pg_version.

When you specify a special pg_version for a particular cluster, you can adjust this parameter at the cluster level accordingly (e.g. some extensions did not exist when PG14 beta was installed).

pg_extensions

PG plugin list, type: string[], level: C, default value:

pg_repack_${pg_version}
pg_qualstats_${pg_version}
pg_stat_kcache_${pg_version}
pg_stat_monitor_${pg_version}
wal2json_${pg_version}"

${pg_version} will be replaced with the major PG version number pg_version.

PG_BOOTSTRAP

On a machine with Postgres, create a set of databases.

  • Cluster identity definition, clean up existing ins, make dir, copy tools and scripts, configure environment variables.
  • Render Patroni config templates, and pull up primary and replica using Patroni.
  • Configure Pgbouncer, initialize the business users and database, and register the database and data source services to DCS.

With pg_conf you can use the default cluster templates (OLTP / OLAP / CRIT / TINY). If you create a custom template, you can clone the default config in roles/postgres/templates and adapt it after modifying. Please refer to customize pgsql cluster for details.

pg_safeguard

Assure that any running pg instance will not be purged by any pgsql playbook., level: C/A, default: false

Check SafeGuard for details.

pg_clean

Remove an existing PostgreSQL instance during initialization? The role fallback is false, while the disposable bundled v1.5.1 pigsty.yml sets pg_clean: true; protected environments should set it to false and enable pg_safeguard.

This allows the removal of any running pg instance during pgsql.yml, which makes it a true idempotent playbook.

It’s a dangerous option so you’d better disable it by default and use it with -e CLI args.

Warning

This parameter not working when pg_safeguard is set to true

pg_data

PG data dir, type: path, level: C, default value: "/pg/data", not recommended to change.

pg_fs_main

PG main data disk mountpoint, type: path, level: C, default value: "/data".

Pigsty’s default dir structure assumes that there is a main data disk mountpoint on the system that holds the DB dir along with another state.

pg_fs_bkup

PG backup disk mountpoint, type: path, level: C, default value: "/data/backups".

Pigsty’s default dir structure assumes that there is a backup data disk mountpoint on the system that holds backup and archive data. However, users can also specify a sub-dir on the primary data disk as the backup disk home mountpoint.

pg_dummy_filesize

Size of the file /pg/dummy, type: size, level: C, default value: "64MiB".

A placeholder file is a pre-allocated empty file that takes up disk space. When the disk is full, removing the placeholder file can free up some space, it is recommended to use 4GiB, and 8GiB for production env.

pg_listen

PG listen IP address, type: ip, level: C, default value: "0.0.0.0".

PG listen to IP address, default all IPv4 0.0.0.0, if you want to include all IPv6, you can use *.

pg_port

PG listen to Port, type: int, level: C, default value: 5432, not recommended to change.

pg_localhost

PG’s UnixSocket dir, type: ip|path, level: C, default value: "/var/run/postgresql".

The Unix socket dir holds the Unix socket files for PostgreSQL and Pgbouncer, which are accessed through the local Unix socket when the client does not specify an IP to access the database.

patroni_enabled

Enabled Patroni, type: bool, level: C, default value: true.

If disable, Pigsty will skip pulling up patroni. This option is used when setting up extra staff for an existing ins.

patroni_mode

Patroni work mode, type: enum, level: C, default value: "default".

  • default: Enable Patroni to enter HA auto-switching mode.
  • pause: Enable Patroni to automatically enter maintenance mode after completing initialization (no automatic M-S S switching).
  • remove: Initialize the cluster with Patroni and remove Patroni after initialization.

pg_dcs_type

Which type of DCS to be used, type: enum, hierarchy: G, default value: "consul".

There are two available options: consul and etcd.

consul_enabled or etcd_enabled should be true if default internal DCS are used.

pg_namespace

DCS namespace used by Patroni, type: path, level: C, default value: "/pg".

patroni_port

Patroni listens to port, type: int, level: C, default value: 8008.

The Patroni API server listens to the port for service and health checks to the public by default.

patroni_watchdog_mode

Patroni Watchdog mode, type: enum, level: C, default value: "automatic".

When an M-S switchover occurs, Patroni will try to shut down the primary before elevating the replica. If the primary is still not shut down within the specified time, Patroni will use the Linux kernel module softdog to fence shutdown according to the config.

  • off: No using watchdog.
  • automatic: Enable watchdog if the kernel has softdog enabled, not forced, default behavior.
  • required: Force watchdog, or refuse to start if softdog is not enabled on the system.

Enabling Watchdog means that the system prioritizes ensuring data consistency and drops availability. If availability is more important to your system, it is recommended to turn off Watchdog on the meta node.

pg_conf

Patroni’s template, type: string, level: C, default value: "tiny.yml"

The Patroni template was used to pull up the Postgres cluster. Pigsty has 4 pre-built templates:

  • oltp.yml Regular OLTP template, default config.
  • olap.ymlOLAP templates to improve parallelism, optimize for throughput, and optimize for long-running queries.
  • crit.yml Core business templates, based on OLTP templates optimized for security, data integrity, using synchronous replication, forced to enable data checksum.
  • tiny.yml Micro templates optimized for low-resource scenarios have demo clusters running in VMs.

pg_libs

Shared database loaded by PG, type: string, level: C, default value: "timescaledb, pg_stat_statements, auto_explain".

Fill in the string of the shared_preload_libraries parameter in the Patroni template to control the dynamic database that PG starts preloading. In the current version, the following databases are loaded by default: timescaledb, pg_stat_statements, auand to_explain.

If Citus support is enabled by default, you need to modify this parameter by adding citus to the first position: citus, timescaledb, pg_stat_statements, auto_explain.

pg_delay

Apply delay for delayed standby cluster, type: interval, level: I, default: 0

Specify a recovery min apply delay for Delayed Replica, can only be set on standby cluster initialization.

pg_checksum

Enable data checksums? , type: bool, class: C , default: "false"

Data checksum is enforced when using crit template.

pg_encoding

PG character set encoding, type: enum, level: C, default value: "UTF8". It is not recommended to modify this parameter if there is no special need.

pg_locale

The locale for PG, type: enum, level: C, default value: "C".

It is not recommended to modify this parameter if there is no special need, improper sorting rules may have a significant impact on database performance.

pg_lc_collate

Collate rule of locale, type: enum, level: C, default value: "C".

Users can implement the localization sorting function by COLLATE expression, wrong localization sorting rule may cause exponential performance loss for some operations, please modify this parameter when you ensure there is a localization requirement.

pg_lc_ctype

C-type of locale, type: enum, level: C, default value: "en_US.UTF8"

Some PG extensions (pg_trgm) require extra character classification definitions to work properly for internationalized characters, so Pigsty will use the en_US.UTF8 character set definition by default, and it is not recommended to modify this parameter.

pgbouncer_enabled

Enable Pgbouncer, type: bool, level: C, default value: true.

pgbouncer_port

Pgbouncer listen port, type: int, level: C, default value: 6432.

pgbouncer_poolmode

Pgbouncer pooling mode, type: enum, level: C, default value: "transaction".

  • transaction, Transaction-level connection pooling, by default, has good performance but affects the use of PreparedStatements with some other session-level features.
  • session, Session-level connection pooling for maximum compatibility.
  • statements, Statement-level join pooling, consider using this pattern if the queries are all point-and-click.

pgbouncer_max_db_conn

Max connection per database, type: int, level: C, default value: 100.

When using Transaction Pooling mode, the number of active server connections is usually in single digits. If Session Pooling mode is used, this parameter can be increased appropriately.

PG_PROVISION

PG_BOOTSTRAP is responsible for creating a completely new set of Postgres clusters, while PG_PROVISION is responsible for creating the default objects in this new set of database clusters, including:

  • Basic roles: read-only role, read-write role, admin role
  • Basic users: replica user, dbsu, monitor user, the admin user
  • Default privileges in the template database
  • Default mode
  • Default Extensions
  • HBA black and white list rules

Pigsty provides rich customization options, if you want to further customize the PG cluster, you can see Customize: PGSQL Cluster.

pg_provision

Provision template to pgsql (app template), type: bool, level: C, default: true.

Provision of the PostgreSQL cluster. Setting to false will skip the tasks defined by pg_provision. Note, however, that the creation of the four default dbsu, replication user, admin user, and monitor user is not affected by this.

pg_init

Custom PG init script, type: string, level: C, default value: "pg-init".

The path to pg-inits Shell script, which defaults to pg-init, is copied to /pg/bin/pg-init and then executed.

The default pg-init is just a wrapper for the SQL command:

# system default roles
psql postgres -qAXwtf /pg/tmp/pg-init-roles.sql

# system default template
psql template1 -qAXwtf /pg/tmp/pg-init-template.sql

# make postgres same as templated database (optional)
psql postgres  -qAXwtf /pg/tmp/pg-init-template.sql

Users can add their cluster init logic in a custom pg-init script.

pg_default_roles

List or global default roles/users, type: role[], level: G/C, default value:

# - default roles - #
pg_default_roles:
  # default roles
  - { name: dbrole_readonly  , login: false , comment: role for global read-only access  }                            # production read-only role
  - { name: dbrole_readwrite , login: false , roles: [dbrole_readonly], comment: role for global read-write access }  # production read-write role
  - { name: dbrole_offline , login: false , comment: role for restricted read-only access (offline instance) }        # restricted-read-only role
  - { name: dbrole_admin , login: false , roles: [pg_monitor, dbrole_readwrite] , comment: role for object creation }  # production DDL change role

  # default users
  - { name: postgres , superuser: true , comment: system superuser }                             # system dbsu, name is designated by `pg_dbsu`
  - { name: dbuser_dba , superuser: true , roles: [dbrole_admin] , comment: system admin user }  # admin dbsu, name is designated by `pg_admin_username`
  - { name: replicator , replication: true , bypassrls: true , roles: [pg_monitor, dbrole_readonly] , comment: system replicator }                   # replicator
  - { name: dbuser_monitor , roles: [pg_monitor, dbrole_readonly] , comment: system monitor user , parameters: {log_min_duration_statement: 1000 } } # monitor user
  - { name: dbuser_stats , password: DBUser.Stats , roles: [dbrole_offline] , comment: business offline user for offline queries and ETL }           # ETL user

This parameter defines the default role and default user in PostgreSQL in the form of an array of objects, which are defined in the same form as in pg_users.

pg_default_privileges

List of default privilegs, type: string[], level: G/C, default value:

pg_default_privileges:
  - GRANT USAGE                         ON SCHEMAS   TO dbrole_readonly
  - GRANT SELECT                        ON TABLES    TO dbrole_readonly
  - GRANT SELECT                        ON SEQUENCES TO dbrole_readonly
  - GRANT EXECUTE                       ON FUNCTIONS TO dbrole_readonly
  - GRANT USAGE                         ON SCHEMAS   TO dbrole_offline
  - GRANT SELECT                        ON TABLES    TO dbrole_offline
  - GRANT SELECT                        ON SEQUENCES TO dbrole_offline
  - GRANT EXECUTE                       ON FUNCTIONS TO dbrole_offline
  - GRANT INSERT, UPDATE, DELETE        ON TABLES    TO dbrole_readwrite
  - GRANT USAGE,  UPDATE                ON SEQUENCES TO dbrole_readwrite
  - GRANT TRUNCATE, REFERENCES, TRIGGER ON TABLES    TO dbrole_admin
  - GRANT CREATE                        ON SCHEMAS   TO dbrole_admin

Please refer to default privilege for details.

pg_default_schemas

List of default schemas, type: string[], hierarchy: G/C, default value: [monitor].

Pigsty creates a schema named monitor for installing monitoring extensions by default.

pg_default_extensions

List of defalut extensions, array of objects, type extension[], hierarchy: G/C, default value:

pg_default_extensions:
  - { name: 'pg_stat_statements',  schema: 'monitor' }
  - { name: 'pgstattuple',         schema: 'monitor' }
  - { name: 'pg_qualstats',        schema: 'monitor' }
  - { name: 'pg_buffercache',      schema: 'monitor' }
  - { name: 'pageinspect',         schema: 'monitor' }
  - { name: 'pg_prewarm',          schema: 'monitor' }
  - { name: 'pg_visibility',       schema: 'monitor' }
  - { name: 'pg_freespacemap',     schema: 'monitor' }
  - { name: 'pg_repack',           schema: 'monitor' }
  - name: postgres_fdw
  - name: file_fdw
  - name: btree_gist
  - name: btree_gin
  - name: pg_trgm
  - name: intagg
  - name: intarray

If the extension does not specify a schema field, the extension will install to the corresponding schema based on the current search_path, e.g., public.

pg_reload

Reload Database Config (HBA), type: bool, level: A, default value: true.

When set to true, Pigsty will execute the pg_ctl reload application immediately after generating HBA rules.

When generating the pg_hba.conf file and manually comparing it before applying it to take effect, you can specify -e pg_reload=false to disable it.

pg_hba_rules

PostgreSQL global HBA rule, type: rule[], hierarchy: G/C, default value:

pg_hba_rules:
  - title: allow meta node password access
    role: common
    rules:
      - host    all     all                         10.10.10.10/32      md5

  - title: allow intranet admin password access
    role: common
    rules:
      - host    all     +dbrole_admin               10.0.0.0/8          md5
      - host    all     +dbrole_admin               172.16.0.0/12       md5
      - host    all     +dbrole_admin               192.168.0.0/16      md5

  - title: allow intranet password access
    role: common
    rules:
      - host    all             all                 10.0.0.0/8          md5
      - host    all             all                 172.16.0.0/12       md5
      - host    all             all                 192.168.0.0/16      md5

  - title: allow local read-write access (local production user via pgbouncer)
    role: common
    rules:
      - local   all     +dbrole_readwrite                               md5
      - host    all     +dbrole_readwrite           127.0.0.1/32        md5

  - title: allow read-only user (stats, personal) password directly access
    role: replica
    rules:
      - local   all     +dbrole_readonly                               md5
      - host    all     +dbrole_readonly           127.0.0.1/32        md5

This parameter is formally identical to pg_hba_rules_extra, and it is recommended to configure a uniform pg_hba_rules globally and use pg_hba_rules_extra for extra customization. The rules in both parameters are applied sequentially, with the latter taking higher priority.

pgbouncer_hba_rules

PgbouncerL global HBA rule, type: rule[], level: G/C, default value:

pgbouncer_hba_rules:
  - title: local password access
    role: common
    rules:
      - local  all          all                                     md5
      - host   all          all                     127.0.0.1/32    md5

  - title: intranet password access
    role: common
    rules:
      - host   all          all                     10.0.0.0/8      md5
      - host   all          all                     172.16.0.0/12   md5
      - host   all          all                     192.168.0.0/16  md5

The default Pgbouncer HBA rules are simple:

  1. Allow login from local with password
  2. Allow password login from the intranet network break

Users can customize it.

PG_EXPORTER

PG Exporter for monitoring Postgres with Pgbouncer connection pools.

pg_exporter_config

PG-exporter config file, type: string, level: C, default value: "pg_exporter.yml".

The default config file used by pg_exporter defines the database and connection pool monitor metrics in Pigsty. The default is pg_exporter.yml.

The PG-exporter config file used by Pigsty is supported by default from PostgreSQL 10.0 and is currently supported up to the latest PG 14 release. There are several of optional templates.

pg_exporter_enabled

Enable PG-exporter, type: bool, level: C, default value: true.

Whether to install and configure pg_exporter, when false, the config of pg_exporter on the current node will be skipped, and this Exporter will be skipped when registering monitoring targets.

pg_exporter_port

PG-exposure listen to Port, type: int, level: C, default value: 9630.

pg_exporter_params

Extra params for PG-exporter URL , type: string, level: C/I, default value: "sslmode=disable".

pg_exporter_url

Monitor target pgurl(override), type: string, level: C/I, default value: "".

The PG URL used by PG-exporter to connect to the database should be the URL to access the postgres managed database, which is configured as an environment variable in /etc/default/pg_exporter.

Optional param, defaults to the empty string, if the pg_exporter_url option is configured, the URL will be used directly as the monitor target pgurl. Otherwise, Pigsty will generate the target URL for monitoring using the following rule:

The above params will be stitched together in the following manner:

postgres://{{ pg_monitor_username }}:{{ pg_monitor_password }}@:{{ pg_port }}/postgres{% if pg_exporter_params != '' %}?{{ pg_exporter_params }}{% if pg_localhost != '' %}&host={{ pg_localhost }}{% endif %}{% endif %}

If the pg_exporter_url param is specified, Exporter will use that connection string directly.

Note: When only a specific business database needs to be monitored, you can use the PGURL of that database directly. if you need to monitor all business databases on a particular database ins, it is recommended to use the PGURL of the meta database postgres.

pg_exporter_auto_discovery

Auto-database-discovery, type: bool, level: C/I, default value: true.

Enable auto-database-discovery, enabled by default. When enabled, PG Exporter automatically detects changes to the list of databases and creates a crawl connection for each database.

When off, monitoring of objects in the library is not available.

Warning

Note that if you have many databases (100+) or a very large number of objects in the database (several k, a dozen), please carefully evaluate the overhead incurred by object monitoring.

pg_exporter_exclude_database

DB auto-discovery exclusion list, type: string, level: C/I, default value: "template0,template1,postgres".

Database name list, when auto-database-discovery is enabled, databases in this list will not be monitored (excluded from monitor objects).

pg_exporter_include_database

Auto-database-discovery capsule list, type: string, level: C/I, default value: "".

Database name list, when auto-database-discovery is enabled, databases that are not in this column table will not be monitored.

pg_exporter_options

Cli args for PG-exporter , type: string, level: C/I, default value:"--log.level=info --log.format=\"logger:syslog?appname=pg_exporter&local=7\"".

pgbouncer_exporter_enabled

Pgbouncer-exporter enabled, type: bool, level: C, default value: true.

pgbouncer_exporter_port

PGB-exporter listens to Port, type: int, level: C, default value: 9631.

pgbouncer_exporter_url

Monitor target pgurl, type: string, level: C/I, default value: "".

The DB’s URL used by PGBouncer Exporter to connect, should be the URL to access the pgbouncer managed database. An optional parameter, default is the empty string.

Pigsty generates the target URL for monitoring by default using the following rules, if the pgbouncer_exporter_url option is configured, this URL will be used directly as the connection string.

PG_EXPORTER_URL='postgres://{{ pg_monitor_username }}:{{ pg_monitor_password }}@:{{ pgbouncer_port }}/pgbouncer?host={{ pg_localhost }}&sslmode=disable'

This option is configured as an environment variable in /etc/default/pgbouncer_exporter.

pgbouncer_exporter_options

Cli args for PGB Exporter, type: string, level: C/I, default value: "--log.level=info --log.format=\"logger:syslog?appname=pgbouncer_exporter&local=7\".

The INFO level log is about to be typed into syslog.

PG_SERVICE

Listen to PostgreSQL service, install the load balancer HAProxy, enable VIP, and configure DNS.

pg_services

Global generic PG service definition, type: service[], level: G, default value:

pg_services:                     # how to expose postgres service in cluster?
  - name: primary                # service name {{ pg_cluster }}-primary
    src_ip: "*"
    src_port: 5433
    dst_port: pgbouncer          # 5433 route to pgbouncer
    check_url: /primary          # primary health check, success when instance is primary
    selector: "[]"               # select all instance as primary service candidate

  - name: replica                # service name {{ pg_cluster }}-replica
    src_ip: "*"
    src_port: 5434
    dst_port: pgbouncer
    check_url: /read-only        # read-only health check. (including primary)
    selector: "[]"               # select all instance as replica service candidate
    selector_backup: "[? pg_role == `primary` || pg_role == `offline` ]"

  - name: default                # service's actual name is {{ pg_cluster }}-default
    src_ip: "*"                  # service bind ip address, * for all, vip for cluster virtual ip address
    src_port: 5436               # bind port, mandatory
    dst_port: postgres           # target port: postgres|pgbouncer|port_number , pgbouncer(6432) by default
    check_method: http           # health check method: only http is available for now
    check_port: patroni          # health check port:  patroni|pg_exporter|port_number , patroni by default
    check_url: /primary          # health check url path, / as default
    check_code: 200              # health check http code, 200 as default
    selector: "[]"               # instance selector
    haproxy:                     # haproxy specific fields
      maxconn: 3000              # default front-end connection
      balance: roundrobin        # load balance algorithm (roundrobin by default)
      default_server_options: 'inter 3s fastinter 1s downinter 5s rise 3 fall 3 on-marked-down shutdown-sessions slowstart 30s maxconn 3000 maxqueue 128 weight 100'

  - name: offline                # service name {{ pg_cluster }}-offline
    src_ip: "*"
    src_port: 5438
    dst_port: postgres
    check_url: /replica          # offline MUST be a replica
    selector: "[? pg_role == `offline` || pg_offline_query ]"         # instances with pg_role == 'offline' or instance marked with 'pg_offline_query == true'
    selector_backup: "[? pg_role == `replica` && !pg_offline_query]"  # replica are used as backup server in offline service

An array consisting of service definition objects that define the services listened to the public. The form is consistent with pg_service_extra.

haproxy_enabled

Enable Haproxy, type: bool, tier: C/I, default value: true.

Pigsty deploys Haproxy on all database nodes by default, enabling Haproxy LB only on specific instance/nodes by overriding ins-level variables.

haproxy_reload

Reload Haproxy config, type: bool, level: A, default value: true.

If turned off, Pigsty will not perform Reload operation after rendering the HAProxy config file, and users can check it by themselves.

haproxy_auth_enabled

Enable auth for Haproxy, type: bool, level: G/C, default value: false.

Not enabled by default, we recommend enabling it in production envs or adding access control to Nginx or other access layers.

haproxy_admin_username

HAproxy admin user name, type: string, level: G, default value: "admin".

haproxy_admin_password

HAproxy admin user password, type: string, level: G, default value: "pigsty".

haproxy_exporter_port

HAproxy-exporter listen port, type: int, tier: C, default value: 9101.

haproxy_client_timeout

HAproxy client timeout, type: interval, level: C, default value: "24h".

haproxy_server_timeout

HAproxy server timeout, type: interval, level: C, default value: "24h".

vip_mode

VIP mode: none, type: enum, level: C, default value: "none".

  • none: No VIP setting, default option.
  • l2: Layer 2 VIP bound to the primary (requires all members to be in the same Layer 2 network broadcast domain).
  • l4: Reserved value for traffic distribution via an external L4 load balancer. (not included in Pigsty’s current implementation).

VIPs are used to ensure the HA of reading and writing services with LBs. When using L2 VIPs, Pigsty’s VIPs are hosted by a vip-manager and will be bound to the cluster primary.

This means that it is always possible to access the cluster primary through a VIP, or the LB on the primary through a VIP (which may have performance pressure).

Note that when using Layer 2 VIP, you must ensure that the VIP candidate ins are under the same Layer 2 network (VLAN, switch).

vip_reload

Overloaded VIP config, type: bool, level: A, default value: true.

vip_address

VIP address used by the cluster, type: string, level: C, default value.

vip_cidrmask

Network CIDR mask length for VIP address, type: int, level: C, default value.

vip_interface

VIP network interface, type: string, level: C, default value: "eth0".

dns_mode

DNS config mode (reserved parameter), type: enum, level: C, default value.

dns_selector

DNS resolution object selector (reserved parameter), type: string, level: C, default value.

30 - Config: REDIS

Historical documentation restored from the Pigsty v1.5.1 tag.

Config Redis cluster and manipulate REDIS playbook behavior. Refer to Redis Deployment for details.

ID Name Section Type Level Comment
700 redis_cluster REDIS_IDENTITY string C redis cluster identity
701 redis_node REDIS_IDENTITY int I redis node identity
702 redis_instances REDIS_IDENTITY instance[] I redis instances definition on this node
710 redis_fs_main REDIS_NODE path C main data disk for redis
711 redis_exporter_enabled REDIS_NODE bool C install redis exporter on redis nodes
712 redis_exporter_port REDIS_NODE int C default port for redis exporter
713 redis_exporter_options REDIS_NODE string C/I default cli args for redis exporter
720 redis_safeguard REDIS_PROVISION bool C set to true to disable purge
721 redis_clean REDIS_PROVISION bool C purge existing redis during init
722 redis_rmdata REDIS_PROVISION bool C remove redis data dir with it?
723 redis_mode REDIS_PROVISION enum C standalone,cluster,sentinel
724 redis_conf REDIS_PROVISION string C which config template will be used
725 redis_bind_address REDIS_PROVISION ip C e.g 0.0.0.0, empty will use inventory_hostname as bind address
726 redis_max_memory REDIS_PROVISION size C/I max memory used by each redis instance
727 redis_mem_policy REDIS_PROVISION enum C memory eviction policy
728 redis_password REDIS_PROVISION string C empty password disable password auth (masterauth & requirepass)
729 redis_rdb_save REDIS_PROVISION string[] C RDB save cmd, disable with empty array
730 redis_aof_enabled REDIS_PROVISION bool C enable redis AOF
731 redis_rename_commands REDIS_PROVISION object C rename dangerous commands
732 redis_cluster_replicas REDIS_PROVISION int C how much replicas per master in redis cluster ?

REDIS_IDENTITY

Identity parameters are the information that must be provided to define a Redis cluster, including:

Name Level Description Example
redis_cluster MUST, cluster level Cluster name redis-test
redis_node MUST, node level Node Number primary, replica
redis_instances MUST, node level Ins Definition { 6001 : {} ,6002 : {}}
  • redis_cluster identifies the Redis cluster name, configured at the cluster level, and serves as the top-level namespace for cluster resources.
  • redis_node identifies the serial number of the node in the cluster.
  • redis_instances is a JSON object with the Key as the ins port and the Value as a JSON object containing the instance-specific config.

redis_cluster

Redis cluster identity, type: string, level: C, default value:

Redis cluster identity will be used as a namespace for resources within the cluster and needs to follow specific naming patterns: [a-z][a-z0-9-]* to be compatible with different constraints on identity identification. It is recommended to use redis- as the cluster name prefix.

Identity param is required params and cluster-level params.

redis_node

Redis node identity, type: int, level: I, default value:

Redis node identity, unique in the cluster, is used to distinguish and identify different nodes, starting with an assignment of 0 or 1.

redis_instances

Redis instances definition on this node, type: instance[], level: I, default value.

This database node deployed all Redis ins in JSON K-V object format. The key is the numeric type port number, and the value is the JSON config entry specific to that instance.

Sample example:

redis_instances: { 6501 : {} ,6502 : {} ,6503 : {} ,6504 : {} ,6505 : {} ,6506 : {} }
redis_instances:
    6501: {}
    6502: { replica_of: '10.10.10.13 6501' }
    6503: { replica_of: '10.10.10.13 6501' }

Each Redis ins listens on a unique port on the node. You can configure separate parameter options for Redis ins (currently, only replica_of is supported for pre-built M-S replication).

Identity params required params and instance-level params.

REDIS_NODE

redis_fs_main

Primary data disk for Redis, type: path, level: C, default value: "/data".

Pigsty will create the redis dir under that dir to store Redis data. For example, /data/redis.

See FHS: Redis for details.

redis_exporter_enabled

Enable Redis exporter, type: bool, level: C, default: true.

Redis Exporter is enabled by default, one on each Redis node deployed and listens on port 9121 by default.

redis_exporter_port

Redis Exporter listens port, type: int, tier: C, default value: 9121.

Note: If you modify this default port, you will need to replace this port along with the relevant config rule file in Prometheus.

redis_exporter_options

Redis Exporter command parameter, type: string, level: C/I, default value: "".

REDIS_PROVISION

redis_safeguard

Disable erasure of existing Redis, type: bool, level: C/A, default value: false.

if true, redis.yml and redis-remove.yml will not remove running redis instance

redis_clean

What to do when Redis exists, type: bool, level: C/A, default value: true.

If true, redis.yml will purge existing instance during init.

redis_rmdata

Remove the Redis data directory together with an instance, type: bool, level: A, default value: true.

When false, residual RDB/AOF files remain and may be loaded by a later instance using the same directory.

redis_mode

Redis cluster mode, type: enum, level: C, default value: "standalone".

Specifies the mode of this Redis cluster, with three optional modes:

  • standalone: Default mode, deploys a series of independent Redis ins.
  • cluster: Redis native cluster mode
  • sentinel: Redis HA component: sentinel

Pigsty also sets up standalone Redis based on the replica_of parameter when using the standalone mode. Pigsty creates a native Redis cluster using all defined instances according to the redis_cluster_replicas parameter when using cluster mode.

redis_conf

Redis config template, type: string, level: C, default value: "redis.conf".

redis_bind_address

Redis listener address, type: ip, level: C, default value: "0.0.0.0".

Redis listener the IP, or inventory_hostname if left blank. The default listener has all local IPv4.

redis_max_memory

Max memory used by each Redis ins, type: size, level: C/I, default value: "1GB"

Max memory used by each Redis ins, default is 1GB; it is recommended to configure this parameter at the cluster level to keep the cluster ins config consistent.

redis_mem_policy

Memory eviction policy, type: enum, level: C, default value: "allkeys-lru".

Other optional policies include:

  • volatile-lru
  • allkeys-lru
  • volatile-lfu
  • allkeys-lfu
  • volatile-random
  • allkeys-random
  • volatile-ttl
  • noeviction

redis_password

Redis password, type: string, level: C, default value: "".

masterauth & requirepass password to use, leave blank to disable password, disabled by default.

Warning

Be careful with security, do not place Redis on the public network without password protection.

redis_rdb_save

RDB SAVE directives, type: string[], level: C, default value: [ "1200 1" ].

Redis SAVE directives, the config will enable RDB functionality, each Save policy as a string。

redis_aof_enabled

Enable AOF, type: bool, level: C, default value: false.

redis_rename_commands

Rename dangerous commands, Type: object, Level: C, Default value: {}.

JSON dictionary renames the command represented by Key to the command represented by Value to avoid misuse of dangerous commands.

redis_cluster_replicas

How many replicas per primary in Redis cluster, type: int, tier: C, default: 1.

/bin/redis-cli --cluster create --cluster-yes \
  --cluster-replicas {{ redis_cluster_replicas|default(1) }}

31 - Customize: PGSQL

Historical documentation restored from the Pigsty v1.5.1 tag.

The Patroni template is used to customize the specification config of the PostgreSQL cluster, while the Postgres template is used to customize the content of the PostgreSQL cluster.

Pigsty provides nearly 100 parameters on PGSQL describing the PostgreSQL cluster.

However, if you profoundly customize the database cluster created by Pigsty, you can see the Patroni template and Postgres template.

Patroni Templates

Pigsty uses Patroni to manage and initialize Postgres clusters. Suppose you wish to modify the default config params, specifications and tuning schemes, HA policies, DCS access, and control APIs of the PostgreSQL cluster. You can do so by modifying the Patroni template.

Pigsty uses Patroni to do the main provisioning work. Even if the user selects no Patroni mode, pulling up the database cluster will be taken care of by Patroni, and removing the Patroni component after the creation is completed.

Users can do most PostgreSQL cluster customization through the Patroni config file. Please refer to Patroni’s official doc for the Patroni config file format.

Predefined Patroni templates

Pigsty provides several predefined initialization templates for initializing the cluster definition files, located by default in roles/postgres/templates/.

Conf CPU Mem Disk Description
oltp 64 400GB 4TB Production OLTP template, default config, optimized latency and performance for production models.
olap 64 400GB 4TB Produce OLAP templates, improve parallelism, optimize for throughput, long queries.
crit 64 400GB 4TB Production core business templates, based on OLTP templates optimized for RPO, security, and data integrity, with synchronous replication and data checksum, enabled.
tiny 1 1GB 40GB Micro templates optimized for low-resource scenarios, such as demo clusters running in virtual machines.
mini 2 4GB 100GB 2C4G model OLTP template
small 4 8GB 200GB 4C8G model OLTP template
medium 8 16GB 500GB 8C16G model OLTP template
large 16 32GB 1TB 16C32G model OLTP template
xlarge 32 64GB 2TB 32C64G model OLTP template

Specify the path to the template to be used via the pg_conf, or fill in the template name if using a predefined template. If a custom Patroni config template is used, the companion node optimization template should also be used for the machine nodes.

pg_conf:   tiny.yml      # Using tiny.yml to tune templates
node_tune: tiny          # Node Tuning Mode：oltp|olap|crit|tiny

During Configure, Pigsty detects the corresponding default specifications that are automatically selected based on the specifications of the current machine (management machine).

Custom Patroni templates

When customizing Patroni templates, you can use several existing templates as a baseline from which to make changes.

Place them in the templates/ dir, just name them in <mode>.yml format.

Please keep the template variables in Patroni, otherwise, the related parameters may not work properly. For example pg_libs.

Finally, in the pg_conf config file, specify the name of your newly created template, e.g., olap-32C128G-nvme.yml.

Postgres templates

The template template1 in the cluster can be customized using the PG template config entry.

This way ensures that any database newly created in that cluster comes with the same default config: schema, extensions, and default privileges.

Related docs

When customizing a template, the relevant parameters are first rendered as SQL scripts to be executed on the deployed cluster.

^---/pg/bin/pg-init
          |
          ^---(1)--- /pg/tmp/pg-init-roles.sql
          ^---(2)--- /pg/tmp/pg-init-template.sql
          ^---(3)--- <other customize logic in pg-init>

# Business users and DB are not created in the template customization
^-------------(4)--- /pg/tmp/pg-user-{{ user.name }}.sql
^-------------(5)--- /pg/tmp/pg-db-{{ db.name }}.sql

pg-init

pg-init is the path to a Shell script for customizing the initialization template that will be executed as a Postgres user, only on the primary, with the primary pulled up at execution. It can run any shell command or any SQL command via psql.

Pigsty will use the default pg-init shell script if this config entry is not specified.

#!/usr/bin/env bash
set -uo pipefail


#==================================================================#
#                          Default Roles                           #
#==================================================================#
psql postgres -qAXwtf /pg/tmp/pg-init-roles.sql


#==================================================================#
#                          System Template                         #
#==================================================================#
# system default template
psql template1 -qAXwtf /pg/tmp/pg-init-template.sql

# make postgres same as templated database (optional)
psql postgres  -qAXwtf /pg/tmp/pg-init-template.sql



#==================================================================#
#                          Customize Logic                         #
#==================================================================#
# add your template logic here

This script can be appended if the user needs to perform complex customization logic. Note pg-init is used to customize database clusters, usually achieved by modifying database templates. When this script is executed, the cluster has been started, but the business users and DB have not yet been created. Therefore the changes to the database templates are reflected in the business database defined by default.

32 - Pigsty Dashboards

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty provides a professional and easy-to-use PostgreSQL monitor system that distills the industry’s monitoring best practices.

The Pigsty monitoring dashboard consists of several relatively independent boards.

Application Description
Home Home
PGSQL PostgreSQL Database Monitor
REDIS Redis Database Monitor
NODES Host Node Monitor
INFRA Infra monitoring/logging
APP Extra added applications

HOME

Pigsty’s home page provides navigation to the various boards.

PGSQL

The PostgreSQL monitoring dashboard has its level, from the top down, as follows：

  • Global: Focuses on the entire environment, the big picture global metrics.
  • Cluster: Focuses on aggregated metrics for a single database cluster.
  • Instance: Focuses on singleton instance objects: database instances, nodes, LBs, and various topic boards.
  • Database (objects): Detailed information about activities, tables, and queries within the database.

Most of the monitoring dashboards can be jumped level through tables and tuples.

Overview Cluster Instance Database
PGSQL Overview PGSQL Cluster PGSQL Instance PGSQL Database
PGSQL Alert PGSQL Service PGSQL Node PGSQL Tables
PGSQL Shard PGSQL Database PGSQL Proxy PGSQL Table
PGSQL MatrixDB PGSQL Replication PGSQL Xacts PGSQL Query
PGSQL Activity PGSQL Queries PGCAT Table
PGSQL Cluster Monly PGSQL Session PGCAT Query
PGCAT Instance PGCAT Database

REDIS

REDIS monitor is divided into a global overview, singleton clusters, and singleton instances.

NODES

NODES monitor is divided into a global overview, single-node clusters, and single nodes.

INFRA

INFRA Monitor is used to monitoring the infra and contains the following Dashboards.

APP

Pigsty comes with a typical application, PGLOG, for analyzing CSV log samples from PG.

Visit https://github.com/vonng/pigsty-app for more sample applications.

33 - Service Discovery

Historical documentation restored from the Pigsty v1.5.1 tag.

This article describes the mechanism used by the Pigsty monitor system Prometheus to discover monitor objects.

The basis of service discovery is identity. For more identity information, please refer to the section entity.

It would help if you also associated monitor targets with the identity in the monitor system, and Pigsty provides two implementations.

The static file is the default service discovery mechanism. Before v1.0.0, Consul was the default service discovery method, and the discovery mechanism could be configured via parameters.

Identity

All instances have an Identity, and the Identifier is the metadata associated with the instance to identify it.

An Identity parameter is a unique identifier that must be defined for any cluster with an instance.

name variables abbreviation type description
cluster pg_cluster cls Core identity parameters Cluster name, top-level namespace for resources within the cluster
role pg_role role core identity parameters instance role, primary, replica, offline,…
seq pg_seq seq core identity parameters Instance serial number, positive integer, unique within the cluster.
instance pg_instance ins derived identity parameters ${pg_cluster}-${pg_seq}
service pg_service svc derived identity parameters ${pg_cluster}-${pg_role}

Attach Identity

After naming the objects in the system, you also need to associate identity information to specific instances.

Identity information is business-given metadata, and the database instance is not aware of this identity information.

The most straightforward way to associate identities is Operator’s memory: the DBA remembers in his mind that the instance on IP 10.2.3.4 is the one used for payments. In contrast, the instance on the other one is used for user management. A better way to manage the identity of cluster members is through the config file or by using service discovery.

Pigsty provides both ways of identity management: service discovery based on the Consul and service discovery based on the config file.

The parameter prometheus_sd_method controls this behavior.

  • consul: service discovery based on Consul, default config.
  • static: service discovery based on local config files.

Pigsty recommends using static service discovery, which is more concise and more reliable as the monitor system does not need to rely on Consul.

Static File Service Discovery

Static file service discovery is the default method of monitor object discovery, and Pigsty pulls the config using the following config by default.

#------------------------------------------------------------------------------
# job: pgsql (database monitoring)
# node_exporter | pg_exporter | pgbouncer_exporter | haproxy(exporter)
# labels: [cls, ins, instance]
# path: targets/pgsql/*.yml
#------------------------------------------------------------------------------
- job_name: pgsql
  metrics_path: /metrics
  file_sd_configs:
    - refresh_interval: 10s
      files: [ /etc/prometheus/targets/pgsql/*.yml ]

The /etc/prometheus/targets dir holds the monitor object definition files generated by Pigsty, and pgsql is the name of the default environment.

Each instance is register by a standalone file:

/etc/prometheus/targets/pgsql
                     ^-----pg-meta-1.yml
                     ^-----pg-test-1.yml
                     ^-----pg-test-2.yml
                     ^-----pg-test-3.yml

Its content is the identifier on a single instance, with monitor objects.

# pg-meta-1 [primary] @ 10.10.10.10
- labels: { cls: pg-meta, ins: pg-meta-1 }
  targets: [10.10.10.10:9630, 10.10.10.10:9100, 10.10.10.10:9631, 10.10.10.10:9101]

Maintenance Document Service Discovery

When using static file service discovery, these config files are automatically maintained for all cluster expansion and downsize.

The config files will be regenerated for all instances in the environment using the following command.

./pgsql.yml -t register_prometheus

Default Targets

Each managed Postgres instance includes several capture ports.

  • Node Exporter for capturing machine node metrics.
  • PG Exporter for capturing database metrics.
  • PGBouncer Exporter for capturing connection pool metrics (uses the same binary as PG Exporter).
  • Patroni for capturing HA components.
  • HAProxy for capturing LB metrics (built-in support, no separate deployment required).

Prometheus on the meta node captures these capture ports.

In addition, the optional Promtail for collecting Postgres, Patroni, and Pgbouncer logs.

All monitoring ports are registered to Consul by default, but Prometheus manages these tasks by default using static file service discovery. Users can use Consul service discovery by configuring prometheus_sd_method as consul to manage instances dynamically.

Consul Service Discovery

Pigsty has built-in DCS-based config management and automatic service discovery, which provides a visual overview of all nodes and services in the system and their health status. All services in Pigsty are automatically registered with DCS.

Users can also use the DNS and service discovery mechanism provided by Consul to achieve automatic DNS-based traffic switching.

Consul uses a Client/Server architecture, with one to five Consul Servers in the entire environment for the actual metadata storage. Consul Agent is deployed on all nodes to proxy the communication between local services and the Consul Server. Pigsty registers services by default using local Consul config files.

Service Registration

A consul agent is running on each node, and services are registered to DCS by the consul agent using JSON config files.

The default location of the JSON config file is /etc/consul.d/, using the naming pattern of svc-<service>.json, taking postgres as an example.

{
  "service": {
    "name": "postgres",
    "port": 5432,
    "tags": [
      "pgsql",
      "primary",
      "pg-meta"
    ],
    "meta": {
      "type": "postgres",
      "role": "primary",
      "seq": "1",
      "instance": "pg-meta-1",
      "service": "pg-meta-primary",
      "cluster": "pg-meta",
      "version": "13"
    },
    "check": {
      "args": ["/usr/pgsql/bin/pg_isready", "-p", "5432", "-U", "dbuser_monitor"],
      "interval": "15s",
      "timeout": "1s"
    }
  }
}

Where the meta and tags sections are the metadata of the service and store the identity information of the instance.

Service Inquiry

Users can discover services registered to Consul through the DNS service provided by Consul, or by calling the Consul API directly.

See Consul doc for ways to consult consul services using the DNS API.

Service Discovery

Prometheus automatically discovers monitor objects in the env via consul_sd_configs. Services tagged with both pg and exporter are automatically identified as crawlers.

- job_name: pg
  # https://prometheus.io/docs/prometheus/latest/configuration/configuration/#consul_sd_config
  consul_sd_configs:
    - server: localhost:8500
      refresh_interval: 5s
      tags:
        - pg
        - exporter

Figure: In services discovered by Prometheus, identity information has been associated with the metric dimension of the instance.

Service Maintenance

A database primary-replica switch occurs, resulting in a registered role that is different from the actual role of the instance. This is when such anomalies need to be handled through the anti-entropy process. Patroni-based failover can fix the registered roles normally through callback logic, but manually done role switchover requires manual intervention to take it. Service registrations to the database can be detected and fixed automatically using the following script. It is recommended to configure Crontab on the database instance or set up periodic patrol tasks on the meta node.

/pg/bin/pg-register $(pg-role)

Labels

Either through Consul or static file service discovery. The end effect is to achieve an association between identity information and instance metrics.

This correlation is achieved through the dimensional labels of the monitoring metrics.

However, all original metrics related to database clusters in Pigsty must have both cls and ins tags and remain immutable throughout their lifecycle.

Identity parameters Dimension labels Sample values
pg_cluster cls pg-test
pg_instance ins pg-test-1
node_ip ip 10.10.10.11

Read the next section of metrics to learn how labels organize these metrics.

34 - Metrics

Historical documentation restored from the Pigsty v1.5.1 tag.

Metric is the core concept of Pigsty’s monitor system.

Format

Metrics are formally cumulative, atomic logical units of measure that can be updated and statistically aggregated over periods.

Metrics typically exist as time series with dimension labels. For example, pg:ins:qps_realtime in the Pigsty sandbox refers to the presentation of real-time QPS for all instances.

pg:ins:xact_commit_rate1m{cls="pg-meta", ins="pg-meta-1", ip="10.10.10.10", role="primary"} 0
pg:ins:xact_commit_rate1m{cls="pg-test", ins="pg-test-1", ip="10.10.10.11", role="primary"} 327.6
pg:ins:xact_commit_rate1m{cls="pg-test", ins="pg-test-2", ip="10.10.10.12", role="replica"} 517.0
pg:ins:xact_commit_rate1m{cls="pg-test", ins="pg-test-3", ip="10.10.10.13", role="replica"} 0

Users can perform operations on indicators: summation, derivation, aggregation, etc.

$ sum(pg:ins:xact_commit_rate1m) by (cls)        -- Query real-time instance QPS aggregated by cluster
{cls="pg-meta"} 0
{cls="pg-test"} 844.6

$ avg(pg:ins:xact_commit_rate1m) by (cls)        -- Query the average real-time instance QPS of all instances in each cluster
{cls="pg-meta"} 0
{cls="pg-test"} 280

$ avg_over_time(pg:ins:qps_realtime[30m])        -- Average QPS of instances in the last 30 minutes
pg:ins:qps_realtime{cls="pg-meta", ins="pg-meta-1", ip="10.10.10.10", role="primary"} 0
pg:ins:qps_realtime{cls="pg-test", ins="pg-test-1", ip="10.10.10.11", role="primary"} 130
pg:ins:qps_realtime{cls="pg-test", ins="pg-test-2", ip="10.10.10.12", role="replica"} 100
pg:ins:qps_realtime{cls="pg-test", ins="pg-test-3", ip="10.10.10.13", role="replica"} 0

Model

Each Metric class of data usually corresponds to multiple time series. Dimensions distinguish different time series corresponding to the same metric.

Metrics + dimension, which can precisely locate a time series. Each time series is an array of (timestamp, fetch) binaries.

Pigsty uses Prometheus’ metrics model, whose logical concept can be represented by the following SQL DDL.

-- Metrics Table,  Metric:TimeSeries = 1:n
CREATE TABLE metrics (
    id   INT PRIMARY KEY,         -- Metrics ID
    name TEXT UNIQUE              -- Metrics Name，[...and other metadata such as type]
);

-- Time Series Table, where each time series corresponds to a metric.
CREATE TABLE series (
    id        BIGINT PRIMARY KEY,               -- Time Series ID
    metric_id INTEGER REFERENCES metrics (id),  -- MetricID which the time series belonged, refer metrics(id)
    dimension JSONB DEFAULT '{}'                -- Dimension information in the form of k-v pair
);

-- Time Series Data table that holds the final sampled data points.
-- Each sampled point belongs to a time series
CREATE TABLE series_data (
    series_id BIGINT REFERENCES series(id),     -- Time Series ID, refer series(id)
    ts        TIMESTAMP,                        -- Timestamp of the data point
    value     FLOAT,                            -- value of the data point
    PRIMARY KEY (series_id, ts)                 -- each data point can be identified by time series id and timestamp
);

Take pg:ins:qps as an example：

-- Sample metric data
INSERT INTO metrics VALUES(1, 'pg:ins:qps');  -- It's a metric named pg:ins:qps, type GAUGE
INSERT INTO series VALUES                     -- The metrics contains 4 time-series, distinguished by dimension labels
(1001, 1, '{"cls": "pg-meta", "ins": "pg-meta-1", "role": "primary", "other": "..."}'),
(1002, 1, '{"cls": "pg-test", "ins": "pg-test-1", "role": "primary", "other": "..."}'),
(1003, 1, '{"cls": "pg-test", "ins": "pg-test-2", "role": "replica", "other": "..."}'),
(1004, 1, '{"cls": "pg-test", "ins": "pg-test-3", "role": "replica", "other": "..."}');
INSERT INTO series_data VALUES                 -- The underneath sampling data point
(1001, now(), 1000),                           -- instance pg-meta-1 qps 1000 at this moment
(1002, now(), 1000),                           -- instance pg-test-1 qps 1000 at this moment
(1003, now(), 5000),                           -- instance pg-test-2 qps 5000 at this moment
(1004, now(), 5001);                           -- instance pg-test-3 qps 5000 at this moment
  • pg_up is a metric with 4-time series, representing the aliveness status of all instances in the sandbox.
  • pg_up{ins": "pg-test-1", ...} is a time series which represent aliveness of the specific instance pg-test-1.

Sources

Pigsty has four primary sources of monitor data: database, connection pool, OS, and LB. Exposed to the public via the corresponding exporter.

Full sources include.

  • PostgreSQL’s monitoring metrics
  • Statistical metrics from the PostgreSQL logs
  • PostgreSQL system directory information
  • Metrics from Pgbouncer connection pool median price
  • PgExporter metrics
  • Metrics of the database working node Node
  • LB Haproxy metrics
  • DCS (Consul) working metrics
  • Monitor system working metrics: Grafana, Prometheus, Nginx
  • Blackbox probing metrics (listed for future coverage in v1.5.1)

Please refer to the v1.5.1 PG exporter definitions section for a complete list of available metrics.

Numbers

Among the database metrics, there are about 230 original metrics related to Postgres and about 50 original metrics related to middleware. Pigsty then designs about 350 DB-related derived metrics based on these actual metrics through hierarchical aggregation and precomputation.

Thus, there are 621 monitor metrics for each database cluster and its attachments for each database cluster. There are 281 machine primitive metrics and 83 derived metrics for a total of 364. Together with the 170 metrics for load balancers, Pigsty has close to 1200 classes of metrics.

Note that here we identify the difference between metric and time-series. We use the term class rather than the individual. This is because a metric may correspond to many time series.

As of 2021, Pigsty’s metrics coverage is one of the best among all open source/commercial monitor systems known to the authors. See Cross-Sectional Comparison for details.

Hierarchy

Pigsty also produces Derived Metrics based on existing metrics.

For example, metrics can be aggregated at different levels.

Entity Identifier Example Label Keys
Environment job pgsql, redis, staging {job}
Shard pg-test-shard\d+ {job, cls*}
Cluster cls pg-meta, pg-test {job, cls}
Service pg-meta-primary, pg-test-replica {job, cls}
Instance ins pg-meta-1, pg-test-1 {job, cls, ins, ip, instance}
Database datname test {..., datname}
Object public.pgbench_accounts {..., datname, <object>}

Take the derived process of TPS metrics as an example.

The original data is the transaction counters captured from Pgbouncer. There are four instances in the cluster and two databases on each instance, so there are eight DB-level TPS metrics for one instance.

The following chart, which is a cross-sectional comparison of QPS for each instance within the entire cluster, uses predefined rules here to first obtain 8 DB-level TPS metrics by deriving the original transaction counters, then aggregating the 8 DB-level time series into four instance-level TPS metrics, and finally aggregating these four instance-level TPS metrics into cluster-level TPS metrics.

Pigsty defines a total of 360 classes of derived aggregated metrics, with more to come. The rules for defining derived metrics are described in Reference: Derived-Metrics.

Special Metric

The catalog is a special indicator.

The boundary between Catalog and Metrics distinction is blurred. For example, the number of pages and the number of tuples in a table, Catalog, or Metrics?

The main difference between Catalog and Metrics in practice is that the information in Catalog is infrequently changed, such as the definition of a table. It would be a waste to grab it once every few seconds like Metrics. So this type of information, which is more static, is classified as Catalog.

The catalog is mainly captured by timed tasks (e.g., Patrol), not Prometheus. Some essential Catalog information, such as some information in pg_class, is also converted to metrics and captured by Prometheus.

Pigsty provides the PGCAT series of monitoring panels to capture and present information directly from the Catalog of the target database.

Summary

After understanding Pigsty metrics, it is helpful to know how Pigsty’s alert system uses these metrics data for practical production purposes.

35 - Alerting

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty uses Prometheus as the primary alerting system.

Alert

Alerts are critical for daily fault response and improving system availability.

Missing alarms will lead to reduced availability, and false alarms will lead to reduced sensitivity, and it is necessary to design the alarm rules prudently.

  • A reasonable definition of alert levels and the corresponding processing flow.
  • A reasonable definition of alert indicators, removal of duplicate alert items, and replenishment of missing alert items.
  • Scientific config of alert thresholds based on historical monitoring data reduces the false alert rate.
  • Reasonably rationalize the special case rules to eliminate false alerts caused by maintenance work, ETL, and offline queries.

Alert Taxonomy

Categorized by source module

  • INFRA: Infra alerts: alerts generated by infra software such as Prometheus, Grafana, Consul, DNS, Nginx, etc.
  • NODES: Node alerts, OS, hardware resources, infra software, load balancing, and other alerts, usually handled by DBA.
  • PGSQL: PostgreSQL alerts, alerts from database/connection pool/load-balancing cluster, usually R&D and DBA concern, DBA handle.
  • REDIS: Redis alerts, R&D and DBA attention, DBA handle.
  • ……： Application board alerts, with alerts by the business side itself, but DBA will set alerts for business metrics like QPS, TPS, Rollback, and Seasonality.
  • Pigsty uses the category tag {infra,pgsql,nodes,redis,....} to identify the level of alerts.

Categorized by urgency

  • P0: CRIT: Incidents with a significant off-site impact require urgent intervention. For example, the primary is down; replication is down. (Incident)
  • P1: WARN: Incidents with minor off-site impact, or incidents with redundant processing, requiring response processing at the minute level. (WARN)
  • P2: INFO: Impending impact, let loose may worsen at the hourly level, requires a response at the hourly level. (Incident)
  • Pigsty uses the severity tag {CRIT, WARN, INFO} to identify the level of urgency of the alert.

Category by Indicator Type

  • Errors: PG Down, PGB Down, Exporter Down, Stream Replication Outage, Single Set Cluster Multi-Master.
  • Traffic: QPS, TPS, Rollback, Seasonality
  • Latency: Average Response Time, Replication Latency
  • Saturation: Connection Stacking, Number of Idle Transactions, CPU, Disk, Age (Transaction Number), Buffer.

Alert Visualization

Pigsty uses timeline status charts to present alert information in various monitor dashboards. The horizontal axis represents a period, and a color bar represents an alert event. Only alerts in the Firing state are displayed in the alert chart; alerts in the Pending state are usually hidden or shown in gray.

Alert Rules

Alert rules can be roughly divided into four types: error, delay, saturation, and traffic.

  • Errors: mainly focus on the aliveness of each component, as well as network outages, brain fractures, and other abnormalities, and the level is usually high (P0|P1).
  • Latency: mainly concerned with query response time, replication latency, slow queries, and long transactions.
  • Saturation: mainly focus on CPU, disk (these two belong to system monitoring but are very important for DB), connection pool queue, number of database back-end connections, age (essentially the saturation of available thing numbers), SSD life, etc.
  • Traffic: QPS, TPS, Rollback (traffic is usually related to business indicators belonging to business monitoring), seasonality of QPS, and a burst of TPS.

Prometheus alert rule

The alert rules are defined using Prometheus syntax, and the full alerting rules are detailed in:

Typical alerts

Errors

A database instance going down will immediately trigger a P0 alert.

# database server down
- alert: PostgresDown
  expr: pg_up < 1
  for: 1m
  labels: { level: 0, severity: CRIT, category: pgsql }
  annotations:
    summary: "CRIT PostgresDown {{ $labels.ins }}@{{ $labels.instance }}"
    description: |
      pg_up[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value }} < 1
      http://g.pigsty/d/pgsql-instance?var-ins={{ $labels.ins }}

When using PostgreSQL in a production environment, the effect of Pgbouncer failure is basically equivalent to Postgres failure, and its survivability alert rule level is unified with Postgres.

# database connection pool down
- alert: PgbouncerDown
  expr: pgbouncer_up < 1
  for: 1m
  labels: { level: 0, severity: CRIT, category: pgsql }
  annotations:
    summary: "CRIT PostgresDown {{ $labels.ins }}@{{ $labels.instance }}"
    description: |
      pgbouncer_up[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value }} < 1
      http://g.pigsty/d/pgsql-instance?var-ins={{ $labels.ins }}

Monitor agent Exporter downtime usually indicates a severe failure: HAProxy and Node Exporter downtime usually means that the LB and database nodes themselves are down and need to be focused on.

#==============================================================#
#                       Agent Aliveness                        #
#==============================================================#
# node & haproxy aliveness are determined directly by exporter aliveness
# including: node_exporter, pg_exporter, pgbouncer_exporter, haproxy_exporter
- alert: AgentDown
  expr: agent_up < 1
  for: 1m
  labels: { level: 0, severity: CRIT, category: infra }
  annotations:
    summary: 'CRIT AgentDown {{ $labels.ins }}@{{ $labels.instance }}'
    description: |
      agent_up[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value  | printf "%.2f" }} < 1
      http://g.pigsty/d/pgsql-alert?viewPanel=22

The duration threshold for all survivability detection is set to 1 minute, which for a 15s acquisition cycle typically means four consecutive failed probes. Regular fast restart operations usually do not trigger survivability alerts.

Cluster brain fracture partitioning

The cluster should have only one partition. If the number of partitions in the cluster is not 1, it means the cluster has entered an abnormal state: unwritable or brain fractured, which will trigger the P0 alarm immediately. Because the detection threshold is 1 minute, so the regular Failover and Switchover usually do not easily trigger this alert.

# cluster partition: split brain
- alert: PostgresPartition
  expr: pg:cls:partition != 1
  for: 1m
  labels: { level: 0, severity: CRIT, category: pgsql }
  annotations:
    summary: "CRIT PostgresPartition {{ $labels.cls }}@{{ $labels.job }} {{ $value }}"
    description: |
      pg:cls:partition[cls={{ $labels.cls }}, job={{ $labels.job }}] = {{ $value }} != 1

Latency

There are two alerts related to replication latency: replication outage, and high replication latency, graded as a P1 warning.

  • Where replication break is an error that is determined using the indicator: pg_downstream_count{state="streaming"}. If the number of replicas in the current streaming state changes negatively, the broken alert is triggered. walsender will determine the replication state, the replica will break directly, and the buffer backlog will go from streaming to catchup state, which will also trigger this alert. Replication interruptions can cause clients to read stale data, which has some off-site impact and is rated as P1.

  • Replication latency can be determined using either latency time or latency bytes. The number of delay bytes is the authoritative indicator. Under normal conditions, replication latency time in 100 milliseconds and replication latency bytes in the order of 100 KB are standard. Based on historical experience data, the time alert threshold of 1MB and 1s is currently used.

#==============================================================#
#                         Replication                          #
#==============================================================#
# replication break for 1m triggers a P1 alert (WARN: heal in 5m)
- alert: PostgresReplicationBreak
  expr: changes(pg_downstream_count{state="streaming"}[5m]) > 0
  # for: 1m
  labels: { level: 1, severity: WARN, category: pgsql }
  annotations:
    summary: "WARN PostgresReplicationBreak: {{ $labels.ins }}@{{ $labels.instance }}"
    description: |
      changes(pg_downstream_count{ins={{ $labels.ins }}, instance={{ $labels.instance }}, state="streaming"}[5m]) > 0


# replication lag bytes > 1MiB or lag seconds > 1s
- alert: PostgresReplicationLag
  expr: pg:ins:lag_bytes > 1048576 or pg:ins:lag_seconds > 1
  for: 1m
  labels: { level: 1, severity: WARN, category: pgsql }
  annotations:
    summary: "WARN PostgresReplicationLag: {{ $labels.ins }}@{{ $labels.instance }}"
    description: |
      pg:ins:lag_bytes[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value | printf "%.0f" }} > 1048576 or
      pg:ins:lag_seconds[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value | printf "%.2f" }} > 1

In addition, there are corresponding alert rules for query latency and disk latency.

For example, the average disk read/write response time lasts more than 32ms for one minute, or the average query RT in Pgbouncer exceeds 16ms will trigger a P1 alert.


# read latency > 32ms (typical on pci-e ssd: 100µs)
- alert: NodeDiskSlow
  expr: node:dev:disk_read_rt_1m > 0.032 or node:dev:disk_write_rt_1m > 0.032
  for: 1m
  labels: { level: 1, severity: WARN, category: node }
  annotations:
    summary: 'WARN NodeReadSlow {{ $labels.ins }}@{{ $labels.instance }} {{ $value  | printf "%.6f" }}'
    description: |
      node:dev:disk_read_rt_1m[ins={{ $labels.ins }}] = {{ $value  | printf "%.6f" }} > 32ms


# pgbouncer avg response time > 16ms (database level)
- alert: PgbouncerQuerySlow
  expr: pgbouncer:db:query_rt_1m > 0.016
  for: 3m
  labels: { level: 1, severity: WARN, category: pgsql }
  annotations:
    summary: "WARN PgbouncerQuerySlow: {{ $labels.ins }}@{{ $labels.instance }} [{{ $labels.datname }}]"
    description: |
      pgbouncer:db:query_rt_1m[ins={{ $labels.ins }}, instance={{ $labels.instance }}, datname={{ $labels.datname }}] = {{ $value | printf "%.3f" }} > 0.016

Saturation

Saturation metrics primary resources, including many system-level monitoring metrics. Mainly includes CPU, disk, connection pool queue, number of database back-end connections, age (essentially saturation of available thing numbers), SSD life, etc.

Database Load

Database load is the combined maximum (percentage, but can exceed 100% when overloaded) of machine CPU usage, Pgbouncer time utilization, Postgres time utilization (14 introduced). Load is the most critical metric in Pigsty, concentrating on the load water level of the database instances and clusters.

#==============================================================#
#                        Saturation                            #
#==============================================================#
# instance pressure higher than 70% for 1m triggers a P1 alert
- alert: PostgresPressureHigh
  expr: ins:pressure1 > 0.70
  for: 1m
  labels: { level: 1, severity: WARN, category: pgsql }
  annotations:
    summary: "WARN PostgresPressureHigh: {{ $labels.ins }}@{{ $labels.instance }}"
    description: |
      ins:pressure1[ins={{ $labels.ins }}, instance={{ $labels.instance }}] = {{ $value | printf "%.3f" }} > 0.70

Queueing Detection

The heap contains two main types of metrics, the number of back-end connections and active connections of the PG on the one hand, and the queuing of the connection pool on the other.

PGB queuing is the decisive metric; it represents that perceptible blocking has occurred on the user side, so the presence of queuing lasting 1 minute triggers a P0 alert.

When using Session Pooling mode, this alert metric can be relaxed appropriately.

# pgbouncer client queue exists
- alert: PgbouncerClientQueue
  expr: pgbouncer:db:waiting_clients > 1
  for: 1m
  labels: { level: 0, severity: CRIT, category: pgsql }
  annotations:
    summary: "CRIT PgbouncerClientQueue: {{ $labels.ins }}@{{ $labels.instance }} [{{ $labels.datname }}]"
    description: |
      pgbouncer:db:waiting_clients[ins={{ $labels.ins }}, instance={{ $labels.instance }}, datname={{ $labels.datname }}] = {{ $value | printf "%.0f" }} > 1

The number of back-end connections is a vital warning metric. If the back-end connections consistently reach the maximum number of connections, it often means an avalanche as well. The number of queued connections in the connection pool also reflects this situation but does not cover the case where the application is directly connected to the database.

Currently, Pigsty uses connection utilization as an alerting metric, i.e., the percentage of available database connections that have been used, with a P1 alert departing if it exceeds 70% for 3 minutes.

# database connection usage > 70%
- alert: PostgresConnUsageHigh
  expr: pg:db:conn_usage > 0.70
  for: 3m
  labels: { level: 1, severity: WARN, category: pgsql }
  annotations:
    summary: "WARN PostgresConnUsageHigh: {{ $labels.ins }}@{{ $labels.instance }} [{{ $labels.datname }}]"
    description: |
      pg:db:conn_usage[ins={{ $labels.ins }}, instance={{ $labels.instance }}, datname={{ $labels.datname }}] = {{ $value | printf "%.3f" }} > 0.70

Idle in Transaction

The number of connections with Idle in Transaction status in the database, more than 2 for 3 minutes, is the departure of the P1 alert.

# database connection usage > 70%
- alert: PostgresIdleInXact
  expr: pg:db:ixact_backends > 1
  for: 3m
  labels: { level: 2, severity: INFO, category: pgsql }
  annotations:
    summary: "Info PostgresIdleInXact: {{ $labels.ins }}@{{ $labels.instance }} [{{ $labels.datname }}]"
    description: |
      pg:db:ixact_backends[ins={{ $labels.ins }}, instance={{ $labels.instance }}, datname={{ $labels.datname }}] = {{ $value | printf "%.0f" }} > 1

Resource Alert

Age (XID) usage exceeds 80% departure P0 alert, which means the system is about to run out of transaction number resources and enter the XID Wraparound state.

# database age saturation > 80%
- alert: PostgresXidWarpAround
  expr: pg:db:age > 0.80
  for: 1m
  labels: { level: 0, severity: CRIT, category: pgsql }
  annotations:
    summary: "CRIT PostgresXidWarpAround: {{ $labels.ins }}@{{ $labels.instance }} [{{ $labels.datname }}]"
    description: |
      pg:db:age[ins={{ $labels.ins }}, instance={{ $labels.instance }}, datname={{ $labels.datname }}] = {{ $value | printf "%.0f" }} > 80%

36 - Install Applications

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty can be used to deploy and monitor PostgreSQL and make and distribute data Applications.

Pigsty provides three sample applications.

  • pglog, which analyzes PostgreSQL CSV log samples.
  • covid, which visualizes WHO COVID-19 data and accesses country outbreak data.
  • pglog, NOAA ISD, allows querying weather observation records from 1901 for 30,000 surface weather stations.

Structure of the application

A Pigsty application typically includes at least one or all of the following.

  • A graphical interface (Grafana Dashboard Definition) placed in the ui dir.
  • Data definitions (PostgreSQL DDL File), placed in the sql dir.
  • Data files (various resources, files to download), placed in the data dir.
  • Logical scripts (executing various types of logic), placed in the bin dir.

A Pigsty application will provide an installation script in the application root dir: install or a shortcut to it. You need to use an admin user to install the meta node. The installation script detects the current environment (gets METADB_URL, PIGSTY_HOME, GRAFANA_ENDPOINT to perform the installation).

Dashboards with the APP label are included in the App drop-down menu in the Pigsty Grafana home page navigation. The home page dashboard navigation includes dashboards with the APP and Overview labels.

You can download the app with the base data from https://github.com/Vonng/pigsty/releases/download/v1.5.1/app.tgz.

COVID

A more straightforward sample data application: visualize WHO COVID-19 data and access country outbreak data.

Public demo: http://demo.pigsty.cc/d/covid-overview

Installation method 

cd covid
make all # Full installation (will download the latest data from WHO)
make all2 # Complete installation (will use the local downloaded data directly)

For finer control.

make ui # install covid dashboards to grafana
make sql # Create covid database table definitions into metadb
make download # Download the latest WHO data
make load # Load the downloaded WHO data
make reload # download + load

If data is already downloaded (e.g., get applications via downloading app.tgz), run make all2 instead to skip the download.

ISD

A feature-complete data application that queries 30,000 surface weather stations worldwide for weather observations from 1901 onwards.

Public demo: http://demo.pigsty.cc/d/isd-overview

Project address: https://github.com/Vonng/isd

Installation 

cd isd
make all # Complete installation (will download the latest data from Github and NOAA)
make all2 # Complete installation (will use the locally downloaded data directly)

For more fine-grained control.

make ui # install covid dashboards to grafana
make sql # Create covid database table definitions into metadb
make download # Download the latest NOAA data, ISD Parser, dictionary tables
make baseline # Initialize the most basic global schema functions with the downloaded data
make reload # Download the latest daily summary from NOAA and parse and load it

PGLOG CSVLOG Sample Analysis

PGLOG Analysis & PGLOG Session provide introspection about PostgreSQL csvlog sample (via table pglog.sample on cmdb).

  • PGLOG Analysis: Analysis of csvlog sample on CMDB (focusing on entire log sample).
  • PGLOG Session: Analysis of csvlog sample (focusing on the single session).

There are some handy alias & func sets on the meta node.

Load csvlog from stdin into sample table.

alias pglog="psql service=meta -AXtwc 'TRUNCATE pglog.sample; COPY pglog.sample FROM STDIN CSV;'"  # useful alias

Get log from pgsql node.

# default: get pgsql csvlog (localhost @ today)
function catlog(){ # getlog <ip|host> <date:YYYY-MM-DD>
    local node=${1-'127.0.0.1'}
    local today=$(date '+%Y-%m-%d')
    local ds=${2-${today}}
    ssh -t "${node}" "sudo cat /pg/data/log/postgresql-${ds}.csv"
}

Combine theme to fetch and load csvlog sample.

catlog | pglog                       # get local (metadb) today's log
catlog node-1 '2021-07-15' | pglog   # get node-1's csvlog @ 2021-07-15

37 - Docker Applications

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty v1.5.1 comes with Docker and Docker Compose deployment support, where Docker Daemon will be enabled by default on the meta node.

You can use Docker to deploy and launch software applications quickly. You can directly access the PostgreSQL/Redis database deployed on the host in the container using the connection string.

  • PgAdmin4: A GUI tool for managing PostgreSQL instances.
  • PGWeb: A tool automatically generates back-end API services based on PG database schema.
  • PostgREST: A tool to automatically generate backend API services based on PG database schema.
  • ByteBase: A GUI tool for making PostgreSQL schema changes.
  • Jupyter Lab: A battery-included Python lab environment for data analysis and processing.

You can also use Docker to execute some battery-included command tools.

  • SchemaSPY: Generates detailed visual reports of database schemas.
  • Pgbadger: Generate database log report.

You can also use Docker to pull up some battery-included open-source SaaS services.

  • Gitlab: open-source code hosting platform.
  • Habour: open-source mirror repo
  • Jira: open-source project management platform.
  • Confluence: open-source knowledge hosting platform.
  • Odoo: open-source ERP
  • Mastodon: PG-based social network
  • Discourse: open-source forum based on PG and Redis

Add Upstream to Nginx

Most of the software described in this article provides a web interface to the public. While it can be accessed directly via IP: Port, we recommend using a domain name and unifying access from the Nginx proxy. Use the following configuration and commands to register a new service with Nginx.

# Add new Nginx services definition
nginx_upstreams:
  - { name: kong         , domain: api.pigsty , endpoint: "127.0.0.1:8880"   } #== v optional ==#
  - { name: pgadmin      , domain: adm.pigsty , endpoint: "127.0.0.1:8885"   }
  - { name: pgweb        , domain: cli.pigsty , endpoint: "127.0.0.1:8886"   }
  - { name: bytebase     , domain: ddl.pigsty , endpoint: "127.0.0.1:8887"   }
  - { name: jupyter      , domain: lab.pigsty , endpoint: "127.0.0.1:8888"   }

./infra.yml -t nginx_config,nginx_restart    # refresh nginx conf, and restart to apply

PgAdmin4

PGAdmin4 is the popular PG control tool; use the following command to pull up the PgAdmin4 service on the meta node, default to host 8885 port, username [email protected], password: pigsty.

cd ~/pigsty/app/pgadmin ; make up
# docker-compose up -d , which is
docker run --init --name pgadmin --restart always --detach --publish 8885:80 \
	-e PGADMIN_DEFAULT_EMAIL=[email protected] -e PGADMIN_DEFAULT_PASSWORD=pigsty \
	dpage/pgadmin4

Copy the server access information to the /tmp/servers.json file and re-import it.

make conf       # provision pgadmin with pigsty pg servers list
make dump       # dump servers.json from pgadmin container

PGWeb

PGWeb is a browser-based PG client tool. Use the following command to pull up the PGWEB service on the meta node, defaulting to the host 8081 port.

cd ~/pigsty/app/pgweb ; docker-compose up -d
docker run --init --name pgweb --restart always --detach --publish 8886:8081 sosedoff/pgweb

Users need to fill in the database connection string, for example, the default CMDB:

postgres://dbuser_dba:[email protected]:5432/meta?sslmode=disable.

PostgREST

PostgREST is a binary component that automatically generates a REST API based on the PostgreSQL database schema.

The following command will pull up postgrest using docker (local port 8082, using the default admin user, exposing the Pigsty CMDB schema).

cd ~/pigsty/app/postgrest ; docker-compose up -d
docker run --init --name postgrest --restart always --detach --publish 8884:8081 postgrest/postgrest

http://home.pigsty.cc:8883/ shows the available API that exposed by PostgREST

Visiting http://10.10.10.10:8884 will show all the definitions of the auto-generated APIs, which can be automatically generated in the Swagger Editor.

curl http://10.10.10.10:8884/pg_cluster will anonymously access the data table pigsty.pg_cluster.

If you want to add, delete, check and design more fine-grained privilege control, please refer to Tutorial 1 - The Golden Key to generate a signed JWT.

ByteBase

ByteBase is a tool for making database schema changes. The following command will start a ByteBase on meta node port 8887.

cd app/bytebase; docker-compose up -d
docker run --init --name bytebase \
        --restart always --detach \
        --publish 8887:8887 \
        --volume /data/bytebase/data:/var/opt/bytebase \
        bytebase/bytebase:1.0.4 \
        --data /var/opt/bytebase \
        --host http://ddl.pigsty \
        --port 8887 \
        --pg postgres://dbuser_bytebase:[email protected]:5432/bytebase

Visit http://10.10.10.10:8887/ to use ByteBase. To start schema changes, you need to create the project, environment, instance, and database.

Jupyter

Jupyter Lab is a data analysis environment. The following command will start a Jupyter Server on port 8084.

docker run -it --restart always --detach --name jupyter -p 8083:8888 -v "${PWD}":/tmp/notebook jupyter/scipy-notebook
docker logs jupyter # Print logs and get Token of login

Visit http://10.10.10.10:8888/ to use JupyterLab, (you need to fill in the auto-generated Token). Note that Pigsty also has JupyterLab installed on the host.

SchemaSPY

Generate a database schema report using CMDB as an example. The following docker, using.

docker run -v /www/schema/pg-meta/meta/pigsty:/output andrewjones/schemaspy-postgres:latest \
    -host 10.10.10.10 -port 5432 -u dbuser_dba -p DBUser.DBA -db meta -s pigsty

Then visit http://pigsty/schema/pg-meta/meta/pigsty to access the Schema report.

Gitlab

Please refer to the Gitlab Docker Deploy Doc to complete the Docker deployment.

export GITLAB_HOME=/data/gitlab

sudo docker run --detach \
  --hostname gitlab.example.com \
  --publish 443:443 --publish 80:80 --publish 23:22 \
  --name gitlab \
  --restart always \
  --volume $GITLAB_HOME/config:/etc/gitlab \
  --volume $GITLAB_HOME/logs:/var/log/gitlab \
  --volume $GITLAB_HOME/data:/var/opt/gitlab \
  --shm-size 256m \
  gitlab/gitlab-ee:latest

sudo docker exec -it gitlab grep 'Password:' /etc/gitlab/initial_root_password

Discourse

Build open source forum Discourse. You need to adjust the config app.yml, focusing on the SMTP part of the config.

Sample Discourse config 
templates:
  - "templates/web.china.template.yml"
  - "templates/postgres.template.yml"
  - "templates/redis.template.yml"
  - "templates/web.template.yml"
  - "templates/web.ratelimited.template.yml"
## Uncomment these two lines if you wish to add Lets Encrypt (https)
# - "templates/web.ssl.template.yml"
# - "templates/web.letsencrypt.ssl.template.yml"
expose:
  - "80:80"   # http
  - "443:443" # https
params:
  db_default_text_search_config: "pg_catalog.english"
  db_shared_buffers: "768MB"
env:
  LC_ALL: en_US.UTF-8
  LANG: en_US.UTF-8
  LANGUAGE: en_US.UTF-8
  EMBER_CLI_PROD_ASSETS: 1
  UNICORN_WORKERS: 4
  DISCOURSE_HOSTNAME: forum.pigsty
  DISCOURSE_DEVELOPER_EMAILS: '[email protected],[email protected]'
  DISCOURSE_SMTP_ENABLE_START_TLS: false
  DISCOURSE_SMTP_AUTHENTICATION: login
  DISCOURSE_SMTP_OPENSSL_VERIFY_MODE: none
  DISCOURSE_SMTP_ADDRESS: smtpdm.server.address
  DISCOURSE_SMTP_PORT: 80
  DISCOURSE_SMTP_USER_NAME: [email protected]
  DISCOURSE_SMTP_PASSWORD: "<password>"
  DISCOURSE_SMTP_DOMAIN: mail.pigsty.cc
volumes:
  - volume:
      host: /var/discourse/shared/standalone
      guest: /shared
  - volume:
      host: /var/discourse/shared/standalone/log/var-log
      guest: /var/log

hooks:
  after_code:
    - exec:
        cd: $home/plugins
        cmd:
          - git clone https://github.com/discourse/docker_manager.git
run:
  - exec: echo "Beginning of custom commands"
  # - exec: rails r "SiteSetting.notification_email='[email protected]'"
  - exec: echo "End of custom commands"

Then, just execute the following command and pull up Discourse.

./launcher rebuild app

Mastodon

The v1.5.1 source reserved this section, but did not ship or document a Mastodon deployment template.

38 - Upgrade Grafana Backend Database

Historical documentation restored from the Pigsty v1.5.1 tag.

You can use postgres as the database used by the Grafana backend.

In this tutorial, you will learn about the following.

TL; DR 

vi pigsty.yml   # uncomment user/db definition：dbuser_grafana  grafana
bin/createuser  pg-meta  dbuser_grafana
bin/createdb    pg-meta  grafana

psql postgres://dbuser_grafana:DBUser.Grafana@meta:5436/grafana -c \
  'CREATE TABLE t(); DROP TABLE t;'    # check pgurl connectivity

vi /etc/grafana/grafana.ini            # edit [database] section: type & url
systemctl restart grafana-server

Create Postgres Cluster

We can define a new database grafana on pg-meta. A Grafana-specific database cluster can also be created on a new machine node: pg-grafana.

Define Cluster

To create a new dedicated database cluster pg-grafana on two bare nodes 10.10.10.11, 10.10.10.12, define it in the config file.

pg-grafana:
  hosts:
    10.10.10.11: {pg_seq: 1, pg_role: primary}
    10.10.10.12: {pg_seq: 2, pg_role: replica}
  vars:
    pg_cluster: pg-grafana
    pg_databases:
      - name: grafana
        owner: dbuser_grafana
        revokeconn: true
        comment: grafana primary database
    pg_users:
      - name: dbuser_grafana
        password: DBUser.Grafana
        pgbouncer: true
        roles: [dbrole_admin]
        comment: admin user for grafana database

Create Cluster

Complete the creation of the database cluster pg-grafana with the following command: pgsql.yml.

bin/createpg pg-grafana # Initialize the pg-grafana cluster

This command calls Ansible Playbook pgsql.yml to create the database cluster.

./pgsql.yml -l pg-grafana # The actual equivalent Ansible playbook command executed

The business users and databases defined in pg_users and pg_databases are created automatically when the cluster is initialized. After creating the cluster using this configuration, the following connection string access database can be used.

postgres://dbuser_grafana:[email protected]:5432/grafana # direct connection to the primary
postgres://dbuser_grafana:[email protected]:5436/grafana # direct connection to the default service
postgres://dbuser_grafana:[email protected]:5433/grafana # Connect to the string read/write service

postgres://dbuser_grafana:[email protected]:5432/grafana # direct connection to the primary
postgres://dbuser_grafana:[email protected]:5436/grafana # Direct connection to default service
postgres://dbuser_grafana:[email protected]:5433/grafana # Connected string read/write service

By default, Pigsty is installed on a single meta node. Then the required users and databases for Grafana are created on the existing pg-meta database cluster instead of using the pg-grafana cluster.

Create Biz User

The convention for business object management is to create users first and then create the database.

Define User

To create a user dbuser_grafana on a pg-meta cluster, add the following user definition to pg-meta’s cluster definition.

Add location: all.children.pg-meta.vars.pg_users.

- name: dbuser_grafana
  password: DBUser.Grafana
  comment: admin user for grafana database
  pgbouncer: true
  roles: [ dbrole_admin ]

If you have defined a different password here, replace the corresponding parameter with the new password.

Create User

Complete the creation of the dbuser_grafana user with the following command.

bin/createuser pg-meta dbuser_grafana # Create the `dbuser_grafana` user on the pg-meta cluster

Calls Ansible Playbook pgsql-createuser.yml to create the user

./pgsql-createuser.yml -l pg-meta -e pg_user=dbuser_grafana # Ansible

The dbrole_admin role has the privilege to perform DDL changes in the database, which is precisely what Grafana needs.

Create Biz Database

Define database

Create business databases in the same way as business users. First, add the definition of the new database grafana to the cluster definition of pg-meta.

Add location: all.children.pg-meta.vars.pg_databases.

- { name: grafana, owner: dbuser_grafana, revokeconn: true }

Create database

Use the following command to complete the creation of the grafana database.

bin/createdb pg-meta grafana # Create the `grafana` database on the `pg-meta` cluster

Calls Ansible Playbook pgsql-createdb.yml to create the database.

./pgsql-createdb.yml -l pg-meta -e pg_database=grafana # The actual Ansible playbook to execute

Access Database

Check Connectivity

You can access the database using different services or access methods.

postgres://dbuser_grafana:DBUser.Grafana@meta:5432/grafana # Direct connection
postgres://dbuser_grafana:DBUser.Grafana@meta:5436/grafana # default service
postgres://dbuser_grafana:DBUser.Grafana@meta:5433/grafana # primary service

We will use the default service that accesses the database directly from the primary through the LB.

First, check if the connection string is reachable and if you have privileges to execute DDL commands.

psql postgres://dbuser_grafana:DBUser.Grafana@meta:5436/grafana -c \
  'CREATE TABLE t(); DROP TABLE t;'

Config Grafana

For Grafana to use the Postgres data source, you need to edit /etc/grafana/grafana.ini and modify the config entries.

[database]
;type = sqlite3
;host = 127.0.0.1:3306
;name = grafana
;user = root
# If the password contains # or ; you have to wrap it with triple quotes. Ex """#password;"""
;password =
;url =

Change the default config entries.

[database]
type = postgres
url = postgres://dbuser_grafana:DBUser.Grafana@meta/grafana

Subsequently, restart Grafana.

systemctl restart grafana-server

See from the monitor system that the new grafana database is already active, then Grafana has started using Postgres as the primary backend database. However, the original Dashboards and Datasources in Grafana have disappeared. You need to re-import Dashboards and Postgres Datasources.

Manage Dashboard

You can reload the Pigsty monitor dashboard by going to the files/ui dir in the Pigsty dir using the admin user and executing grafana.py init.

cd ~/pigsty/files/ui
./grafana.py init # Initialize the Grafana monitor dashboard using the Dashboards in the current directory

Execution results in:

vagrant@meta:~/pigsty/files/ui
$ ./grafana.py init
Grafana API: admin:pigsty @ http://10.10.10.10:3000
init dashboard : home.json
init folder pgcat
init dashboard: pgcat / pgcat-table.json
init dashboard: pgcat / pgcat-bloat.json
init dashboard: pgcat / pgcat-query.json
init folder pgsql
init dashboard: pgsql / pgsql-replication.json
init dashboard: pgsql / pgsql-table.json
init dashboard: pgsql / pgsql-activity.json
init dashboard: pgsql / pgsql-cluster.json
init dashboard: pgsql / pgsql-node.json
init dashboard: pgsql / pgsql-database.json
init dashboard: pgsql / pgsql-xacts.json
init dashboard: pgsql / pgsql-overview.json
init dashboard: pgsql / pgsql-session.json
init dashboard: pgsql / pgsql-tables.json
init dashboard: pgsql / pgsql-instance.json
init dashboard: pgsql / pgsql-queries.json
init dashboard: pgsql / pgsql-alert.json
init dashboard: pgsql / pgsql-service.json
init dashboard: pgsql / pgsql-persist.json
init dashboard: pgsql / pgsql-proxy.json
init dashboard: pgsql / pgsql-query.json
init folder pglog
init dashboard: pglog / pglog-instance.json
init dashboard: pglog / pglog-analysis.json
init dashboard: pglog / pglog-session.json

This script detects the current environment (defined at ~/pigsty during installation), gets Grafana access information, and replaces the URL connection placeholder domain name (*.pigsty) in the monitor dashboard with the real one in use.

export GRAFANA_ENDPOINT=http://10.10.10.10:3000
export GRAFANA_USERNAME=admin
export GRAFANA_PASSWORD=pigsty

export NGINX_UPSTREAM_YUMREPO=yum.pigsty
export NGINX_UPSTREAM_CONSUL=c.pigsty
export NGINX_UPSTREAM_PROMETHEUS=p.pigsty
export NGINX_UPSTREAM_ALERTMANAGER=a.pigsty
export NGINX_UPSTREAM_GRAFANA=g.pigsty
export NGINX_UPSTREAM_HAPROXY=h.pigsty

As a reminder, using grafana.py clean will clear the target monitor dashboard, and using grafana.py load will load all the monitor dashboards in the current dir. When Pigsty’s monitor dashboard changes, you can use these two commands to upgrade all the monitor dashboards.

Manage DataSources

When creating a new PostgreSQL cluster with pgsql.yml or a new business database with pgsql-createdb.yml, Pigsty will register the new PostgreSQL data source in Grafana, and you can access the target database instance directly through Grafana using the default admin user. Most of the functionality of the application pgcat relies on this.

To register a Postgres database, you can use the register_grafana task in pgsql.yml.

./pgsql.yml -t register_grafana # Re-register all Postgres data sources in the current environment
./pgsql.yml -t register_grafana -l pg-test # Re-register all the databases in the pg-test cluster

Update Grafana Database

You can directly change the backend data source used by Grafana by modifying the Pigsty config file. Edit the grafana_database and grafana_pgurl parameters in pigsty.yml and change them.

grafana_database: postgres
grafana_pgurl: postgres://dbuser_grafana:DBUser.Grafana@meta:5436/grafana

Then re-execute the grafana task in infral.yml to complete the Grafana upgrade.

./infra.yml -t grafana

39 - Jupyter Lab

Historical documentation restored from the Pigsty v1.5.1 tag.

TL;DR 

./infra-jupyter.yml # install jupyter on meta node on port 8888 with user jupyter and password pigsty
./infra-jupyter.yml -e jupyter_port=8887 # use another port 8887 by default
./infra-jupyter.yml -e jupyter_username=osuser_jupyter jupyter_password=pigsty2 # use another password

Jupyter Config

ID Name Section Type Level Comment
220 jupyter_port JUPYTER int G jupyter server listen port
221 jupyter_username JUPYTER bool G os user for jupyter lab
222 jupyter_password JUPYTER bool G password for jupyter lab

Jupyter Lab is a complete data science R&D env based on IPython Notebook for data analysis and visualization. It is currently an optional Beta feature and is only enabled in the demo by default.

Because JupyterLab provides a Web Terminal feature, it is recommended to use infra-jupyter to deploy it manually on the meta node.

Default Values 

jupyter_username: jupyter       # os user name, special names: default|root (dangerous!)
jupyter_password: pigsty        # default password for jupyter lab (important!)
jupyter_port: 8887              # default port for jupyter lab

jupyter_port

Jupyter server listen port, type: int, level: G, default value: 8888.

When JupyterLab is enabled, Pigsty will run the local Notebook server using the user-specified by the jupyter_username parameter.

In addition, you need to make sure that the config node_packages_meta_pip parameter contains the default value 'jupyterlab'.

JupyterLab can be accessed by navigating from the Pigsty home page or through the default domain lab.pigsty, and listens on port 8888.

jupyter_username

The OS user used by Jupyter, type: bool, level: G, default value: "jupyter".

The same goes for other usernames, but the particular username default will run Jupyter Lab with the user currently running the installation (usually admin).

jupyter_password

Password for Jupyter Lab, type: bool, level: G, default value: "pigsty".

If Jupyter is enabled, it is highly recommended to change this password. Salted and obfuscated passwords are written to ~jupyter/.jupyter/jupyter_server_config.json by default.

Jupyter Playbook

infra-jupyter

Playbook infra-jupyter.yml will install JupyterLab on the meta node.

It’s a handy data analysis IDE for python. It’s also risky because of its web shell functionality. So it’s disabled by default. And enabled only in the Demo environment.

Refer to Config: Jupyter for configuring Jupiter, then execute this playbook.

Warning

If Jupyter is enabled in the production environment, be sure to change the password of Jupyter.

40 - Backup and Recovery

Historical documentation restored from the Pigsty v1.5.1 tag.

Backup & Recovery matters.

Failures can be divided into two categories: hardware/resource failures and software/human errors.

Replication aims for the former, while delayed replica & cold backups aim for the latter.

Pigsty has complete support for backups. There is battery-included physical replication that can be configured by simply declaring replica & delayed roles. And handy cold backup scripts for cold backup.

Physical Replica

In Pigsty, physical backups are created by specifying roles (pg_role) for the database instances. For example, the following configuration declares a HA database cluster with one primary & two replicas.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary } # Primary
    10.10.10.12: { pg_seq: 2, pg_role: replica } # Hot standby
    10.10.10.13: { pg_seq: 3, pg_role: offline } # Warm standby
  vars:
    pg_cluster: pg-test

Hot Standby

replica = Hot Standby, which carries read-only traffic and maintains real-time synchronization with the primary, with a few replication delays.

It is consistent with the primary and will take over the work of the primary when it fails, and will also take over online read-only traffic. A hot standby that uses sync replication to keep up with the primary in real-time can also be called a sync backup. Under normal circumstances, the latency of physical replication can be in the range of 1ms-100ms / tens of KB to several MB, depending on the network conditions and load level.

Please refer to Classic Physical Replication.

Warm Standby

offline = Warm Standby, warm standby, does not carry online traffic. Backup, or for offline/analysis queries only.

Please refer to offline deployment.

Sync Standby

standby = Sync Standby. Strict real-time sync with the primary.

Use sync commit replica, also called sync standby. Please refer to sync standby deployment for details.

Delayed Replica

Delayed is a quick measure of software failure/human error. Changes are received in real-time from the primary using the standard primary-replica stream replication mechanism but are delayed for a specific period (e.g., one hour, a day) before the application is executed. Thus, it is a copy of the historical state of the original primary. When there is a problem like mistaken data deletion, the delay provides a time window to salvage: immediately query the data from the delayed and backfill the original primary.

A delayed replica can be created using the function standby cluster. For example, now you want to specify a delayed replica for the pg-test cluster: pg-testdelay, which is the state of pg-test 1 hour ago. If there is a mis-deletion of data, it can be immediately retrieved from the delayed and poured back into the original cluster.

# pg-test is the original database
pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
  vars:
    pg_cluster: pg-test
    pg_version: 14

# pg-testdelay will be used as a delayed for the pg-test
pg-testdelay:
  hosts:
    10.10.10.12: { pg_seq: 1, pg_role: primary , pg_upstream: 10.10.10.11 } # The actual role is Standby Leader
  vars:
    pg_cluster: pg-testdelay
    pg_version: 14

After creation, edit the Patroni config file for the delayed cluster using pg edit-config pg-testdelay in the meta node and change standby_cluster.recovery_min_apply_delay to the delay value you expect.

 standby_cluster:
   create_replica_methods:
   - basebackup
   host: 10.10.10.11
   port: 5432
+  recovery_min_apply_delay: 1h

Cold Backup

Cold backup is the final safeguard for your data assets.

The cold backup database exists as a static file of the data-dir and is a binary backup of the database dir. Cold backups are the last resort in case of accidental deletion of databases or tables, or catastrophic failure of the whole cluster/whole server room.

Pigsty provides a script for making cold backups pg-backup, which can be executed as dbsu on the database node to create a full physical backup of the current instance and place it in the /pg/backup (by default located in {{ pg_fs_bkup }}/backup).

With parameters, you can specify the backup database URL, backup-dir, file name, encryption method, retention policy for existing backups, etc.

$ pg-backup                 # Execute the backup script without any arguments
[2021-08-05 17:41:35][INFO] ================================================================
[2021-08-05 17:41:35][INFO] [INIT] pg-backup begin, checking parameters
[2021-08-05 17:41:35][DEBUG] [INIT] #====== BINARY
[2021-08-05 17:41:35][DEBUG] [INIT] pg_basebackup     :   /usr/pgsql/bin/pg_basebackup
[2021-08-05 17:41:35][DEBUG] [INIT] openssl           :   /bin/openssl
[2021-08-05 17:41:35][DEBUG] [INIT] #====== PARAMETER
[2021-08-05 17:41:35][DEBUG] [INIT] filename  (-f)    :   backup_pg-meta_20210805.tar.lz4
[2021-08-05 17:41:35][DEBUG] [INIT] src       (-s)    :   postgres:///
[2021-08-05 17:41:35][DEBUG] [INIT] dst       (-d)    :   /pg/backup
[2021-08-05 17:41:35][DEBUG] [INIT] tag       (-t)    :   pg-meta
[2021-08-05 17:41:35][DEBUG] [INIT] key       (-k)    :   pg-meta
[2021-08-05 17:41:35][DEBUG] [INIT] encrypt   (-e)    :   false
[2021-08-05 17:41:35][DEBUG] [INIT] upload    (-u)    :   false
[2021-08-05 17:41:35][DEBUG] [INIT] remove    (-r)    :   -mmin +1200
[2021-08-05 17:41:35][INFO] [LOCK] acquire lock @ /tmp/backup.lock
[2021-08-05 17:41:35][INFO] [LOCK] lock acquired success on /tmp/backup.lock, pid=25438
[2021-08-05 17:41:35][INFO] [BKUP] backup begin, from postgres:/// to /pg/backup/backup_pg-meta_20210805.tar.lz4
[2021-08-05 17:41:35][INFO] [BKUP] backup in normal mode
pg_basebackup: initiating base backup, waiting for checkpoint to complete
pg_basebackup: checkpoint completed
pg_basebackup: write-ahead log start point: 0/6B000028 on timeline 1
pg_basebackup: write-ahead log end point: 0/6B000138
pg_basebackup: syncing data to disk ...
pg_basebackup: base backup completed
[2021-08-05 17:41:45][INFO] [BKUP] backup complete!
[2021-08-05 17:41:45][INFO] [RMBK] remove local obsolete backup: 1200
[2021-08-05 17:41:45][INFO] [BKUP] find obsolete backups: find /pg/backup/ -maxdepth 1 -type f -mmin +1200 -name 'backup*.lz4'
[2021-08-05 17:41:45][WARN] [BKUP] remove obsolete backups:
[2021-08-05 17:41:45][INFO] [RMBK] remove old backup complete
[2021-08-05 17:41:45][INFO] [LOCK] release lock @ /tmp/backup.lock
[2021-08-05 17:41:45][INFO] [DONE] backup procdure complete!
[2021-08-05 17:41:45][INFO] ================================================================

This script will use pg_basebackup to initiate a backup from the specified PGURL (default is the local database instance), using a tar archive with lz4 compression and optional openssl RC4 stream encryption.

The backup file is placed in the /pg/backup/ dir by default, and the default file name consists of a prefix, cluster name, and date, e.g., backup_pg-meta_20210805.tar.lz4.

The default backup cleanup policy is to clean up old backup files 1200 minutes (20 hours old) when the latest backup completes.

Restoring from cold backup

To use this backup, you need to set the cluster to maintenance mode (pt pause), stop the data cluster primary, and empty the dataset cluster dir. Then the backup file is unpacked to /pg/data.

# Find the latest backup file and print the information
backup_dir="/pg/backup"
data_dir=/pg/data
backup_latest=$(ls -t ${backup_dir} | head -n1)
echo "backup ${backup_latest} will be used"

# Suspend Patroni, shut down the database, and remove the data directory (dangerous)
pg pause pg-meta
pg_ctl -D /pg/data stop
rm -rf /pg/data/*                                     # Emptying the data directory (dangerous)

# Unzip the backup to the database directory
echo "unlz4 -d -c ${backup_dir}/${backup_latest} | tar -xC ${data_dir}"
unlz4 -d -c ${backup_dir}/${backup_latest} | tar -xC ${data_dir}    # Unzip to the database directory
# Optional: If the password is set when encrypting, you need to decrypt it before decompressing it
openssl enc -rc4 -d -k ${PASSWORD} -in ${backup_latest} | unlz4 -d -c | tar -xC ${data_dir}

# Pull up the database again
systemctl restart patroni

# Redo other replicas of the cluster
pg reinit <cluster> # Reset the other instance members of the cluster in turn

There are other handy tools can be used for manage backups: pg_backrest & pg_probackup.

41 - Offline Installation

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty downloads all dependent packages from the Internet and creates local Yum repo during initialization.

The total size of all the dependent software is about 1GB or so. Although Pigsty has tried to use mirror repos as much as possible to speed up the download, the download of a small number of packages may still be blocked by firewalls and may appear very slow. You can set the download proxy to complete the first download by using the proxy_env config entry.

If you are using an OS other than CentOS 7.8, it is recommended that you use the full online download and installation process. And cache the downloaded software after the first initialization is complete. See Making an offline pkg.

If you wish to skip the extended download process, or if the execution control meta-node does not have Internet access, consider downloading a pre-packaged offline pkg.

Contents

To quickly pull up Pigsty, it is recommended to use the offline package and upload method to complete the installation.

The offline pkg includes all packages from the local Yum repo. Pigsty is installed at Infra Init when the local Yum repo is created by default.

{{ nginx_home }}
  |---- {{ repo_name }}.repo
  ^---- {{ repo_name}}/repo_complete
  ^---- {{ repo_name}}/**************.rpm

By default, {{ nginx_home }} is the root dir of the Nginx static file server, which defaults to /www, and repo_name is a custom local source name, which defaults to pigsty.

The /www/pigsty dir contains all RPM packages, and the offline pkg is actually a zip archive of the /www/pigsty dir.

The principle of the offline pkg is that Pigsty checks if the local Yum repo-related files already exist during the execution of the infra initialization. If they already exist, download the package, and its dependencies are skipped.

The marker file used for the check is {{ repo_home }}/{{ repo_name }}/repo_complete, which defaults to /www/pigsty/repo_complete. If this marker file exists, it means that the local YUM repo has been created. Otherwise, Pigsty will perform the usual download logic. Once the download is complete, you can archive a compressed copy of the dir for accelerating the initialization of other environments.

The marker file used for the check is {{ nginx_home }}/{{ repo_name }}/repo_complete, by default /www/pigsty/repo_complete, if this marker file exists, (usually set by Pigsty after the local source is created), then the local source has created and can be used directly. Otherwise, Pigsty will perform the usual download logic. Once the download is complete, you can archive a compressed copy of the directory, which can be used to speed up the initialization of other environments.

Sandbox Environment

Downloading offline installers

Pigsty comes with a sandbox. The offline installer for the sandbox is placed in the files dir by default and can be downloaded from the Github Release page.

curl -SL https://github.com/pgsty/pigsty/releases/download/v1.5.1/pkg.tgz -o dist/v1.5.1/pkg.tgz

Pigsty’s official CDN also provides the latest version of pkg.tgz for download. Just execute the following command.

make download
curl http://download.pigsty.cc/v1.5.1/pkg.tgz -o files/pkg.tgz

Upload offline pkg

When using the Pigsty sandbox, after downloading the offline pkg to the dir of the local files, you can directly upload the offline pkg to the meta node using the shortcut command make copy-pkg provided by Makefile.

Using make upload will also copy the local offline pkg (Yum cache) to the meta node.

# upload rpm cache to meta controller
upload:
	ssh -t meta "sudo rm -rf /tmp/pkg.tgz"
	scp -r files/pkg.tgz meta:/tmp/pkg.tgz
	ssh -t meta "sudo mkdir -p /www/pigsty/; sudo rm -rf /www/pigsty/*; sudo tar -xf /tmp/pkg.tgz --strip-component=1 -C /www/pigsty/"

Make offline pkg

When using the Pigsty sandbox, you can make an offline pkg from the cache of meta nodes in the sandbox by make cache and copying it locally.

# cache rpm packages from meta controller
cache:
	rm -rf pkg/* && mkdir -p pkg;
	ssh -t meta "sudo tar -zcf /tmp/pkg.tgz -C /www pigsty; sudo chmod a+r /tmp/pkg.tgz"
	scp -r meta:/tmp/pkg.tgz files/pkg.tgz
	ssh -t meta "sudo rm -rf /tmp/pkg.tgz"

Prepare pkg.tgz for Production

Before using an offline pkg in a production environment, you must ensure that the OS of the production environment is the same as the OS on which the offline pkg was made. Pigsty uses CentOS 7.8 by default.

If you need to run Pigsty on other versions of OS (e.g., CentOS 7.3, 7.7), it is recommended that you perform the initialization process thoroughly in a sandbox with the same version of the operating system installed, without using the offline pkg and by downloading directly from the upstream repo.

After the regular initialization is completed, users can make the software cache for a specific OS as an offline package by make cache or manually executing the relevant commands.

To build an offline installer package from a local meta node that has completed initialization.

tar -zcf /tmp/pkg.tgz -C /www pigsty # Make an offline package

Using an offline pkg in a production environment is similar to a sandbox. You need to copy pkg.tgz to the meta node and then unzip the offline pkg to the target address.

Here, take the default /www/pigsty as an example. To extract all the contents of the zip package (RPM package, repo_complete markup file, meta DB of repodata source, etc.) to the target dir /www/pigsty, you can use the following command.

mkdir -p /www/pigsty/
sudo rm -rf /www/pigsty/*
sudo tar -xf /tmp/pkg.tgz --strip-component=1 -C /www/pigsty/

42 - CMDB Usage

Historical documentation restored from the Pigsty v1.5.1 tag.

Instead of a static config file, you can use postgres as an inventory for Pigsty.

Using CMDB as a dynamic Inventory for Ansible has several advantages: metadata is presented as data tables in a highly structured way, and database constraints ensure consistency. The CMDB also allows you to use third-party tools to manage Pigsty metadata.

Currently, Pigsty’s CMDB only supports PostgreSQL clusters. If your pigsty.yml contains Redis and MatrixDB, it will report an error. It is recommended to use a separate pigsty.yml config file to manage Redis and Greenplum clusters.

The Pigsty CMDBmode is automatically created during the initialization of the pg-meta meta DB (files/cmdb.sql) and is located in the meta database’s pigsty mode of the meta DB. Static config files can be loaded into the CMDB using bin/inventory_load.

Warning

You must execute infra.yml entirely in the meta node after installation before you can use CMDB.

usage: inventory_load [-h] [-p PATH] [-d CMDB_URL]

load config arguments

optional arguments:
  -h, --help            show this help message and exit
  -p PATH, --path PATH  config path, ${PIGSTY_HOME}/pigsty.yml by default
  -d DATA, --data DATA  postgres cmdb pgurl, ${METADB_URL} by default

By default, executing the script without parameters will load $PIGSTY_HOME/pigsty.yml into the CMDB under the name pgsql.

bin/inventory_load # load default config to default cmdb
bin/inventory_load -p files/conf/pigsty-demo.yml
bin/inventory_load -p files/conf/pigsty-dcs3.yml -d postgresql://dbuser_meta:[email protected]:5432/meta

CMDB as Inventory

Once the original config file is loaded into the CMDB as the initial data, Ansible can be configured to use the CMDB as the inventory.

bin/inventory_cmdb

You can switch back to a static config file.

bin/inventory_conf

Modifying the inventory is essentially a matter of editing ansible. cfg in the Pigsty dir.

---
inventory = pigsty.yml
+++
inventory = inventory.sh

43 - Migration Tutorial

Historical documentation restored from the Pigsty v1.5.1 tag.

An auxiliary playbook, pgsql-migration.yml, provides a battery-included migration method based on logical replication.

By filling in the information about the source and host clusters, the playbook will automatically create the scripts needed for the migration and simply execute them in sequence during the database migration.

activate                        # activate migration context
check-replica-identity          # prepare: make sure all table have replica identity
check-replica-identity-solution # prepare: fix table without replica identity
check-special-objec             # prepare: check special object: matrialized view
compare            # compare: fast check on data consistency (by row count)
copy-schema        # migration: copy schema from src to dst cluster
create-pub         # migration: create publication on source cluster
create-sub         # migration: build logical replication between src & dst clusters
progress           # migration: print logical replication progress
copy-seq           # migration: copy sequence number from src to dst cluster
next-seq           # migration: advance dst cluster by 10000 to fix primary confliction
remove-sub         # remove subscription from dst cluster

Prepare

SRC and DST Clusters

Suppose you want to migrate the pg-meta cluster in the sandbox (containing the Pigsty meta DB with the pgbench test tables) to the pg-test cluster.

pg-meta-1	10.10.10.10  --> pg-test-1	10.10.10.11 (10.10.10.12,10.10.10.13)

First, create a new empty target cluster pg-test, then edit the variables list in pgsql-migration.yml and fill in the relevant information (connection information for the host cluster’s primary).

#--------------------------------------------------------------#
#                   MIGRATION CONTEXT                          #
#--------------------------------------------------------------#

# src cluster (the old cluster)
src_cls: pg-meta                       # src cluster name
src_db: meta                           # src database name
src_ip: 10.10.10.10                    # ip address of src cluster primary
src_list: [ ]                          # ip address list of src cluster members (non-primary)

#--------------------------------------------------------------#
# dst cluster (the new cluster)
dst_cls: pg-test                       # dst cluster name
dst_db: test                           # dst database name
dst_ip: 10.10.10.11                    # dst cluster leader ip addressh
dst_list: [ 10.10.10.12, 10.10.10.13 ] # dst cluster members (non-primary)

# dst cluster access information
dst_dns: pg-test                       # dst cluster dns records
dst_vip: 10.10.10.3                    # dst cluster vip records

#--------------------------------------------------------------#
# credential (assume .pgpass viable)
pg_admin_username: dbuser_dba          # superuser @ both side
pg_replicatoin_username: replicator    # repl user @ src to be used
migration_context_dir: ~/migration     # this dir will be created
#--------------------------------------------------------------#

Execute pgsql-migration.yml, which by default creates the ~/migration/pg-meta.meta dir on the meta node, containing the resources and scripts used for the migration.

Manual Template

Announcement

  • Operation Notice
  • Business Party Notification

Preparations

  • Prepare source and host clusters
  • Repair Source HBA
  • Create Source Replication User
  • External Resource Request
  • Create Cluster Profile
  • Configure business users
  • Configure business database
  • Configure business whitelist
  • Create business cluster
  • Fix Replication Identity
  • Identify migration target
  • Generate schema synchronization command
  • Generate serial number synchronization command
  • Generate create publish command
  • Generate create subscription command
  • Generate progress check command
  • Generate check command

Stock Migration

  • Synchronize database schema
  • Create publish at the source
  • Create a subscription to host
  • Wait for logical replication sync

Switch moment

  • Prepare
  • Stop source write traffic
  • Synchronize sequence numbers with other objects
  • Verify data consistency
  • Flow Switching
  • Aftercare

44 - PGSQL SOP: Standard Operating Procedure

Historical documentation restored from the Pigsty v1.5.1 tag.

Most cluster management operations require using the admin user on the meta node and the execution of the corresponding Ansible Playbook in the Pigsty root dir.

Unless otherwise specified, the following examples demonstrate a three-node cluster pg-test in a sandbox.

Cheatsheet

Cluster Admin

Manage PostgreSQL clusters and instances by executing the following commands on the meta node using the admin user.

# Cluster creation/cluster expansion
./pgsql.yml -l pg-test # Create cluster: initialize pg-test cluster on new machine
./pgsql.yml -l 10.10.10.13 # Add instance (expansion), initialize 10.10.10.13 node in pg-test cluster

# Cluster destruction/instance destruction
./pgsql-remove.yml -l pg-test # Cluster destruction: destroy the pg-test cluster, destroy all non-master instances first, and destroy the master instance last
./pgsql-remove.yml -l 10.10.10.13 # Instance destruction (shrinkage): destroy the 10.10.10.13 nodes in the pg-test cluster

# Business database/user creation
./pgsql-createuser.yml -l pg-test -e pg_user=test # Create a user named test in the pg-test cluster
./pgsql-createdb.yml -l pg-test -e pg_database=test # Create a database named test in the pg-test cluster

# Cluster membership adjustment
./pgsql.yml -l pg-test -t pg_hba # Adjust cluster HBA rules and apply
./pgsql.yml -l pg-test -t haproxy_config,haproxy_reload # Adjust cluster load balancer config and apply

# Service registration information adjustment
./pgsql.yml -l pg-test -t register_prometheus # Register the cluster as a monitoring target to the Prometheus of the meta node
./pgsql.yml -l pg-test -t register_grafana # Register the cluster as a data source to Grafana on the meta node

Patroni Admin

Pigsty uses Patroni to manage PostgreSQL instance databases by default. You need to use the patronictl command to manage Postgres clusters, including cluster config changes, restarts, Failover, Switchover, redoing specific instances, switching automatic/manual HA mode, etc.

Users can use patronictl to manage all database clusters as postgres on the meta node, with alias pt already created on all hosted machines: alias pt='patronictl -c /pg/bin/patroni.yml'.

The commonly used management commands are shown below. For more orders, please refer to pg --help.

pg list [cluster] 				# Print cluster information
pg edit-config [cluster] 		# Edit the config file for a cluster

pg reload [cluster] [instance] 	# reload the config of a cluster or instance
pg restart [cluster] [instance] # Restart a cluster or instance
pg reinit [cluster] [instance] 	# reset an instance in a cluster (recreate the replica)

pg pause [cluster] 				# enter maintenance mode (does not trigger automatic failover, Patroni no longer operates Postgres)
pg resume [cluster]		 		# exit maintenance mode

pg failover [cluster] 			# Manually trigger Failover for a cluster
pg switchover [cluster] 		# Manually trigger a Switchover for a cluster

Component Admin

In Pigsty deployments, all components are managed by systemd; except for PostgreSQL, which Patroni manages.

Exception: exception when patroni_mode is remove, Pigsty will use systemd to manage Postgres directly.

systemctl stop patroni               # Close Patroni & Postgres
systemctl stop pgbouncer             # Close Pgbouncer
systemctl stop pg_exporter           # Close PG Exporter
systemctl stop pgbouncer_exporter    # Close Pgbouncer Exporter
systemctl stop node_exporter         # Close Node Exporter
systemctl stop haproxy               # Close Haproxy
systemctl stop vip-manager           # Close Vip-Manager
systemctl stop consul                # Close Consul
systemctl stop postgres              # Close Postgres (Use only when patroni_mode = remove)

The following components can be reloaded via systemctl reload.

systemctl reload patroni             # Overload config: Patroni
systemctl reload postgres            # Overload config: Postgres （Use only when patroni_mode = Remove ）
systemctl reload pgbouncer           # Overload config:  Pgbouncer
systemctl reload pg_exporter         # Overload config: PG Exporter
systemctl reload pgbouncer_exporter  # Overload config:  Pgbouncer Exporter
systemctl reload haproxy             # Overload config:  Haproxy
systemctl reload vip-manager         # Overload config:  vip-manager
systemctl reload consul              # Overload config: Consul

The config of the infra components can also be reloaded via systemctl reload on the meta node.

systemctl reload nginx          # Overload config: nginx (update the index of haproxy management interface and external access domain name)
systemctl reload prometheus     # Overload config: Prometheus (update pre calculation index calculation logic and alarm rules)
systemctl reload alertmanager   # Overload config: Alertmanager
systemctl reload grafana-server # Overload config： Grafana

When Patroni manages Postgres, do not use pg_ctl to manipulate the database cluster (/pg/data).

After entering maintenance mode via pg pause <cluster>, you can manually manage the database.

Common Tasks 

./infra.yml -t environ             # Re-configure environment variables and access credentials on the meta node
./infra.yml -t repo_upstream       # Re-add the upstream repo to the meta node
./infra.yml -t repo_download       # Re-download the package on the meta node
./infra.yml -t nginx_home          # Regenerate the Nginx home page content
./infra.yml -t nginx_config,nginx_restart # Regenerate the Nginx config file and restart
./infra.yml -t prometheus_config   # Reset Prometheus config
./infra.yml -t grafana_provision   # Reset the Grafana monitoring dashboard
./pgsql.yml -l pg-test -t=pgsql    # Complete database deployment: database, monitoring, services
./pgsql.yml -l pg-test -t=postgres # Complete database deployment
./pgsql.yml -l pg-test -t=monitor  # Complete the monitoring deployment
./pgsql.yml -l pg-test -t=service  # Complete load balancing deployment, (Haproxy & VIP)
./pgsql.yml -l pg-test -t=register # Register the service to the infrastructure
./pgsql.yml -l pg-test -t=register # Register the service to the infrastructure
./pgsql.yml -l pg-test -t=consul   # Reset the DCS server, you need to configure the cluster to maintenance mode first

Case 1: Cluster Create and Expand

Cluster create/expand uses the playbook pgsql.yml to create a cluster using the cluster name as the execution object and to create a new instance/cluster expand using a single instance in the cluster as the execution object.

Cluster Creation 

./nodes.yml -l pg-test      # Initialize the machine nodes contained in pg-test
./pgsql.yml -l pg-test      # Initialize the pg-test database cluster

The above two playbooks can be simplified as follows.

bin/createpg pg-test

Cluster Expansion

Suppose you have a test cluster pg-test with two instances 10.10.10.11 and 10.10.10.12, and now you expand one additional 10.10.10.13.

Modify config

First, you need to modify the corresponding config in the inventory (pigsty.yml or CMDB).

Warning

Please make sure to note that the pg_seq must be unique for each instance in the cluster.

pg-test:
  hosts:
    10.10.10.11: { pg_seq: 1, pg_role: primary }
    10.10.10.12: { pg_seq: 2, pg_role: replica }
    10.10.10.13: { pg_seq: 3, pg_role: replica, pg_offline_query: true } # New Forces
  vars: { pg_cluster: pg-test }

Execute changes

Then, execute the following command to complete the initialization of cluster members.

./nodes.yml -l 10.10.10.13      # Initialize the pg-test machine node 10.10.10.13
./pgsql.yml -l 10.10.10.13      # Initialize the pg-test instance pg-test-3

# The above two commands can be simplified as follows:
bin/createpg 10.10.10.13

Adjusting Roles

Cluster expansion will result in changes in cluster membership. Please refer to Case 8: Cluster Role Adjustment to distribute the traffic to the new instance.

Frequently Asked Questions

FAQ 1: Database and Consul already exist, execution aborted

Pigsty uses a SafeGuard to avoid accidental deletion of running instances; please use the pgsql-remove playbook to complete the instance destruction first and then reuse the node. For an emergency overwrite installation, you can use the following parameters to force the running instance to be erased during the installation (Danger!!!)

For example: ./pgsql.yml -l pg-test -e pg_clean=true will force an override install of the pg-test cluster.

When the execution of a Consul instance is aborted, you can also take the above parameters to force the running instance to be erased.

FAQ 2: The database is too extensive, waiting for the replica to come online timeout

When an expansion operation gets stuck at the Wait for postgres replica online step and aborts, it is usually because the existing database instance is too large and exceeds Ansible’s timeout wait time.

If you abort with an error, the instance will continue to pull up the replica instance in the background. You can use the pg list pg-test command to list the current status of the cluster, and when the status of the new replica is running, you can use the following command to continue the Ansible Playbook from where it was aborted.

./pgsql.yml -l 10.10.10.13 --start-at-task 'Wait for postgres replica online'

Another way is to directly and explicitly specify subsequent tasks.

./pgsql.yml -l 10.10.10.13 -t pg_hba,pg_patroni,pgbouncer,pg_user,pg_db,monitor,service,register

If pulling up a new replica node is aborted due to some accident, please refer to FAQ 1.

FAQ 3: The cluster is in maintenance mode, and the replica is not automatically pulled up.

Solution 1: Use pg resume pg-test to configure the cluster in auto switchover mode and perform the replica creation operation.

Solution 2, use pg reinit pg-test pg-test-3 to manually complete the instance initialization. This command can also be used to redo existing instances in the cluster.

FAQ 4: Cluster replica with clonefrom tag, but not suitable for use or pull failed due to data corruption

Find the problem machine, switch to postgres user, modify the patroni config file and reload it to take effect。

sudo su postgres
sed -ie 's/clonefrom: true/clonefrom: false/' /pg/bin/patroni.yml
sudo systemctl reload patroni
pg list -W # Check the cluster status and confirm that the failed instance does not have the clonefrom tag

FAQ 5: How to create a fixed admin user using an existing user

By default, the system uses dba as the admin user, which should be able to ssh into the remote database node and execute sudo commands password-free from the admin machine.

If the assigned machine does not have this user by default, but you have another admin user (e.g., vagrant) that can ssh into the remote node and execute sudo, you can run the following command to log into the remote machine using the other user and automatically create the standard admin user.

./nodes.yml -t node_admin -l pg-test -e ansible_user=vagrant -k -K
SSH password:
BECOME password[defaults to SSH password]:

If you specify the -k|--ask-pass -K|--ask-become-pass parameter, you should enter the admin user’s SSH login password and sudo password before executing.

Once executed, you can log in to the target database machine from the admin user on the meta node (default dba) and run other playbooks.

FAQ 6: Cluster replica with clonefrom tag, but not usable or failed to pull due to data corruption

Find the machine in question, switch to the postgres user, modify the patroni config file and reload it to take effect.

sudo su postgres
sed -ie 's/clonefrom: true/clonefrom: false/' /pg/bin/patroni.yml
sudo systemctl reload patroni
pg list -W # Check the cluster status and confirm that the failed instance does not have the clonefrom tag

Case 2: Cluster Destruction and Downsize

Cluster destruction/downsize uses a dedicated playbook pgsql-remove that, when used against a cluster, will take the entire cluster destruction.

When used against a single instance in the cluster, the instance will be removed from the cluster.

Note that removing the cluster primary directly will cause the cluster to Failover, so please remove all replicas first when removing instances one by one.

Warning

Note that the pgsql-remove playbook is not affected by the security insurance parameter and will remove the database instance and the cluster directly, so please use it carefully!

Cluster Destruction 

# Destroy the pg-test cluster: destroy all non-primary instances first and the primary instance last
./pgsql-remove.yml -l pg-test

# When destroying the cluster, remove the data dir and packages together
./pgsql-remove.yml -l pg-test -e rm_pgdata=true -e rm_pgpkgs=true

# Remove the nodes contained in pg-test, optionally
./nodes-remove.yml -l pg-test

Cluster Downsize 

./pgsql-remove.yml -l 10.10.10.13  # Instance destruction (shrinkage): destroy the 10.10.10.13 node in the pg-test cluster
./nodes-remove.yml -l 10.10.10.13  # Remove 10.10.10.13 nodes from Pigsty (optional)

Adjustment of roles

Note: Cluster downsizing will result in a change in cluster membership. When downsizing, the health check of this instance is false, and the traffic carried initially by this instance will be immediately transferred to other members. However, you still need to refer to the instructions in Reference Case 8: PGSQL Role Adjustment to completely remove this offline instance from the cluster config.

Downline Offline Instance

Note that in the default config, if an instance with pg_role = offline or pg_offline_query = true is taken offline, only the primary instance remains in the cluster. Then there will be no instances left to carry offline read traffic.

Case 3: Cluster Config Change and Restart

Cluster config modification

Modifying the PostgreSQL cluster config needs to be done via pg edit-config <cluster>, especially for the synchronous replication option synchronous_mode, which must be changed in the Patroni config entry (.synchronous_mode), not (postgresql. parameters.synchronous_mode and other parameters).

After the config is saved, configs that do not require a restart can take effect by confirmation.

Please note that the parameters modified by pg edit-config are cluster parameters. The config parameters in the scope of individual instances (e.g., Patroni’s Clonefrom tag, etc.) need to be modified directly in the Patroni config file (/pg/bin/patroni.yml), and systemctl reload patroni to take effect.

Warning

Please note that HBA rules are created automatically by Pigsty. Please do not use Patroni to manage HBA rules.

Cluster reboot

Configs that require a restart then need to schedule a database restart. Restarting the cluster can be done with the following command.

pg restart [cluster] [instance] # Restart a cluster or instance

The pending restart notation is displayed in the ``pg list ` with the instances that need to be restarted to take effect.

Case 4: Create PGSQL Biz User

A new business user can be created in an existing database via pgsql-createuser.yml.

Business users are usually those used by software programs in a production environment, and users who need to access the database through connection pools must be managed in this way. Other users can be created and managed using Pigsty or can be maintained and managed.

# Create a user named test in the pg-test cluster
./pgsql-createuser.yml -l pg-test -e pg_user=test

The above command can be abbreviated as:

bin/createuser pg-test test # Create a user named test in the pg-test cluster

If you need to create both the business user and the business database, you should usually create the business user first.

If the database is configured with an OWNER, create the corresponding OWNER user first and then make the corresponding database.

Case 5: Create PGSQL BIZ DB

A new business database can be created in an existing database cluster by pgsql-createdb.yml.

A business database refers to a database object that is created and used by a user. If you wish to access this database through a connection pool, it must be created using the playbook provided by Pigsty to maintain the config in the connection pool consistent with PostgreSQL.

# Create a database named test in the pg-test cluster
./pgsql-createdb.yml -l pg-test -e pg_database=test

The above command can be abbreviated as:

bin/createdb pg-test test # Create a database named test in the pg-test cluster

If the database is configured with an OWNER, please create the corresponding OWNER user first before creating the corresponding database.

Register the new database as a Grafana data source

Executing the following command will register all the business databases on all instances in the pg-test cluster into Grafana as PostgreSQL data sources for use by the PGCAT application.

./pgsql.yml -t register_grafana -l pg-test

Case 6: APPLY PGSQL HBA

Users can adjust the HBA config of an existing database cluster/instance via the pg_hba subtask of pgsql.yml.

This task should be re-executed when the cluster undergoes Failover, Switchover, and HBA rule adjustments to adjust the cluster’s IP black and white list rules to the expected behavior.

Pigsty strongly recommends using config files to automatically manage HBA rules unless you know exactly what you do.

The HBA config is generated by combining pg_hba_rules with pg_hba_rules_extra, both of which are arrays of rule config objects. The sample example is as follows.

- title: allow internal infra service direct access
  role: common
  rules:
    - host putong-confluence     dbuser_confluence     10.0.0.0/8  md5
    - host putong-jira           dbuser_jira           10.0.0.0/8  md5
    - host putong-newjira        dbuser_newjira        10.0.0.0/8  md5
    - host putong-gitlab         dbuser_gitlab         10.0.0.0/8  md5

The following command will regenerate the HBA rule and apply it to take reloadable.

./pgsql.yml -t pg_hba -l pg-test

The above command can be abbreviated as follows:

bin/reloadhba pg-test

Case 7: PGSQL LB Traffic Control

HAProxy controls the cluster traffic of PostgreSQL in Pigsty by default, and users can control the cluster traffic directly through the WebUI provided by HAProxy.

Controlling traffic using HAProxy Admin UI

Pigsty’s HAProxy provides an Admin UI on port 9101 (haproxy_exporter_port by default, which can be accessed by default via Pigsty’s default domain name suffixed with the instance name (pg_ cluster-pg_seq) to access it. The admin UI comes with optional auth options enabled by the parameter (haproxy_auth_enabled). Admin interface auth is not enabled by default, and when enabled, it is required to use the username specified by haproxy_admin_username and haproxy_admin_password with the username and password to log in.

Use your browser to access http://pigsty/<ins> (the domain name varies by configuration, you can also click there from the PGSQL Cluster Dashboard) to access the LB admin interface on the corresponding instance. Sample Interface

Here you can control the traffic of one service per set of masses and each back-end server. For example, you can select that Server to drain the corresponding Server, set the MAINT state, and apply it. If you are using multiple HAProxy for load balancing simultaneously, you will need to perform this action on each LB in turn.

Modify Cluster Configuration

When a cluster changes its members, you should adjust the load balancing config of all cluster members at the appropriate time to faithfully reflect the cluster architecture changes, such as when a primary-replica switch occurs.

In addition, by configuring the pg_weight parameter, you can explicitly control the percentage of load carried by each instance in the cluster. The change requires regenerating the HAProxy config file in the cluster and reloading the reload to take effect. For example, this config reduces the relative weight of instance 2 in all services from the default of 100 to 0.

10.10.10.11: { pg_seq: 1, pg_role: primary}
10.10.10.12: { pg_seq: 2, pg_role: replica, pg_weight: 0 }
10.10.10.13: { pg_seq: 3, pg_role: replica,  }

Use the following command to adjust the cluster config and take effect.

# Regenerate the HAProxy config for pg-test (but not applied)
./pgsql.yml -l pg-test -t haproxy_config

# Reload the HAProxy config for pg-test and enable it to take effect
./pgsql.yml -l pg-test -t haproxy_config -e haproxy_reload=true

The config and enable commands can be combined and abbreviated as follows:

bin/reloadha pg-test  # Adjusting all HAPROXY for the pg-test cluster and reloading the config usually does not affect existing traffic.

Case 8: PGSQL Role Adjustment

This describes Pigsty’s default HAProxy access method, which may be different if you are using L4 VIP or other access methods.

This adjustment is required when any kind of role change occurs in the cluster. The pg_role parameter of the cluster and instance in the inventory does not truly reflect the server state.

For example, when a cluster is scaled downsize, cluster load balancing immediately redistributes traffic based on health checks but does not remove config entries for downstream instances.

After cluster expansion, the LB config of existing instances will not change. You can access all current cluster members via HAProxy on the new instance. Still, the HAProxy config on the old instance remains unchanged, so no traffic is distributed to the new instance.

1. Modify the config file pg_role

When a primary-replica switch of the cluster has occurred, the pg_role of the cluster members should be adjusted according to the current actual situation. For example, when pg-test has a Failover or Switchover that causes the pg-test-3 instance to become the new primary, you should modify the role of pg-test-3 to primary and configure the original primary pg_role to the replica.

Also, you should ensure that at least one instance in the cluster can be used to provide Offline services, so configure the instance parameter for pg-test-1: pg_offline_query: true. It is highly discouraged to configure more than one Offline instance for a cluster, as slow queries and long transactions may cause online read-only traffic to suffer.

10.10.10.11: { pg_seq: 1, pg_role: replica, pg_offline_query: true }
10.10.10.12: { pg_seq: 2, pg_role: replica }
10.10.10.13: { pg_seq: 3, pg_role: primary }

2. Adjusting cluster instance HBAs

The HBA rules that apply to different roles should also be returned when the cluster role changes.

Use the method described in Case 6: APPLY PGSQL HBA to adjust the cluster HBA rules

3. Adjusting the cluster load balancing config

HAProxy dynamically distributes request traffic based on the health check results returned by Patroni in the cluster, so node failure does not affect external requests. However, users should adjust the cluster load balancing config at the right time (e.g., after waking up in the morning). For example, take the failure out of the cluster config entirely instead of continuing to freeze in the cluster with a health check DOWN status.

Use the method described in Case 7: PGSQL LB Traffic Control to tune the cluster load balancing config.

4. Consolidation Operations

You can use the following commands after modifying the config to complete the tuning of the cluster roles.

./pgsql.yml -l pg-test -t pg_hba,haproxy_config,haproxy_reload

Or use the equivalent abbreviated script.

bin/reloadhba pg-test # Adjust cluster HBA config
bin/reloadha pg-test  # Tune cluster HAProxy config

Case 9: Monitor Targets

Pigsty manages Prometheus monitor objects by default using static file service discovery, default location: /etc/prometheus/targets.

Using Consul service discovery is optional, and in this mode, there is usually no need to manage monitor objects manually. When using static file service discovery, all monitor objects are automatically handled together with the execution instance when it goes online and offline: registered or logged out. However, some particular scenarios cannot be fully covered (e.g., changing cluster names).

Adding Prometheus monitor objects manually.

# Register all members of the pg-test cluster as prometheus monitor objects
./pgsql.yml -t register_prometheus -l pg-test

PostgreSQL service discovery object definitions are stored by default in the /etc/prometheus/targets/pgsql dir of all managed nodes. Each instance corresponds to a yml file containing the target’s label, with the port exposed by the Exporter.

# pg-meta-1 [primary] @ 10.10.10.10
- labels: { cls: pg-meta, ins: pg-meta-1, ip: 10.10.10.10 }
  targets: [10.10.10.10:9630, 10.10.10.10:9631, 10.10.10.10:9101, 10.10.10.10:8008]

Manually remove Prometheus monitor objects.

# Remove the monitor object file
rm -rf /etc/prometheus/targets/pgsql/pg-test-*.yml

Add Grafana data source manually

# Register each database object in the pg-test cluster as a grafana data source
./pgsql.yml -t register_grafana -l pg-test

Remove Grafana data source manually

In Grafana, click Data Source Management and manually remove it.

Case 10: Cluster Switchover

For example, if you want to perform a Failover on the three-node demo cluster pg-test, you can execute the following command.

pg failover <cluster>

Then follow the wizard prompts to execute Failover. After cluster Failover, you should refer to the instructions in Case 8: Cluster Role Adjustment to fix the cluster role.

Execute Failover's operation log 
[08-05 17:00:30] postgres@pg-meta-1:~
$ pg list pg-test
+ Cluster: pg-test (6988888117682961035) -----+----+-----------+-----------------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Pending restart | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+-----------------+
| pg-test-1 | 172.21.0.3  | Leader  | running |  1 |           |                 | clonefrom: true |
| pg-test-2 | 172.21.0.4  | Replica | running |  1 |         0 | *               | clonefrom: true |
| pg-test-3 | 172.21.0.16 | Replica | running |  1 |         0 | *               | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+-----------------+

[08-05 17:00:34] postgres@pg-meta-1:~
$ pg failover pg-test
Candidate ['pg-test-2', 'pg-test-3'] []: pg-test-3
Current cluster topology
+ Cluster: pg-test (6988888117682961035) -----+----+-----------+-----------------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Pending restart | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+-----------------+
| pg-test-1 | 172.21.0.3  | Leader  | running |  1 |           |                 | clonefrom: true |
| pg-test-2 | 172.21.0.4  | Replica | running |  1 |         0 | *               | clonefrom: true |
| pg-test-3 | 172.21.0.16 | Replica | running |  1 |         0 | *               | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+-----------------+
Are you sure you want to failover cluster pg-test, demoting current master pg-test-1? [y/N]: y
2021-08-05 17:00:46.04144 Successfully failed over to "pg-test-3"
+ Cluster: pg-test (6988888117682961035) -----+----+-----------+-----------------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Pending restart | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+-----------------+
| pg-test-1 | 172.21.0.3  | Replica | stopped |    |   unknown |                 | clonefrom: true |
| pg-test-2 | 172.21.0.4  | Replica | running |  1 |         0 | *               | clonefrom: true |
| pg-test-3 | 172.21.0.16 | Leader  | running |  1 |           | *               | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+-----------------+

[08-05 17:00:46] postgres@pg-meta-1:~
$ pg list pg-test
+ Cluster: pg-test (6988888117682961035) -----+----+-----------+-----------------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Pending restart | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+-----------------+
| pg-test-1 | 172.21.0.3  | Replica | running |  2 |         0 | *               | clonefrom: true |
| pg-test-2 | 172.21.0.4  | Replica | running |  2 |         0 | *               | clonefrom: true |
| pg-test-3 | 172.21.0.16 | Leader  | running |  2 |           | *               | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+-----------------+

Case 11: Reset Component

As the saying goes, a reboot can solve 90% of the problems, while reinstallation can solve the remaining 10%.

Resetting the problem component is a simple and effective means of stopping it. Using Pigsty’s initialization playbook infra.yml with pgsql.yml can reset the infra with the database cluster, but usually, we only need to use specific subtasks to reset particular components.

Infrastructure Reset

Common infra reconfig commands include.

./infra.yml -t repo_upstream       # Re-add the upstream repo to the meta node
./infra.yml -t repo_download       # Re-download the package on the meta node
./infra.yml -t nginx_home          # Regenerate the Nginx home page content
./infra.yml -t prometheus_config   # Reset Prometheus configuration
./infra.yml -t grafana_provision   # Reset the Grafana monitoring dashboard

You can also forcibly reinstall these components.

./infra.yml -t nginx      # Reset Nginx
./infra.yml -t prometheus # Reset Prometheus
./infra.yml -t grafana    # Reset Grafana
./infra-jupyter.yml       # Reset Jupyterlab

In addition, you can reset specific components on the database node using the following command.

# The more commonly used, safe reset command, re-install monitoring and re-registration will not affect the service
./pgsql.yml -l pg-test -t=monitor  # Redeploy monitoring
./pgsql.yml -l pg-test -t=register # Re-register the service to the infrastructure (Nginx, Prometheus, Grafana, CMDB...)
./nodes.yml -l pg-test -t=consul -e dcs_clean=true # Reset DCS Agent in maintenance mode

# A slightly risky reset operation
./pgsql.yml -l pg-test -t=service   # Redeploy load balancing, may cause service to flash off
./pgsql.yml -l pg-test -t=pgbouncer  # Redeploy connection pooling, may cause service to flash

# Very dangerous reset task
./pgsql.yml -l pg-test -t=postgres # Reset databases (including Patroni, Postgres, Pgbouncer)
./pgsql.yml -l pg-test -t=pgsql    # Redo the complete database deployment: database, monitoring, services
./nodes.yml -l pg-test -t=consul   # Reset DCS server directly when HA auto-switchover mode is enabled

# Extremely dangerous reset task
./nodes.yml -l pg-test -t=consul -e dcs_clean=true -e dcs_safeguard=false # Force wipe DCS servers, may cause all DB clusters to be unwritable

For example, if there is a problem with the cluster’s connection pool, a default way to stop the damage is to restart or reinstall the Pgbouncer connection pool.

./pgsql.yml -l pg-test -t=pgbouncer # reinstall the connection pool (all users and DBs will be regenerated), manually modified config will be lost

Case 12: Switching DCS Servers

DCS (Consul/Etcd) is a very reliable service.

According to Patroni’s working logic, once the cluster primary finds that the DCS server is unreachable, it will immediately follow the Fencing logic and downgrade itself to a normal replica, unable to write.

Maintenance Mode

Unless the cluster is currently in “maintenance mode” (enter with pg pause <cluster> and exit with pg resume <cluster>).

# Put the target cluster into maintenance mode
pg pause pg-test

# Restore the target cluster to automatic failover mode (optional)
pg resume pg-test

Reset DCS service for PGSQL Nodes

When DCS fails to be available, and you need to migrate to a new DCS (Consul) cluster, you can use the following actions.

First, create the new DCS cluster, then edit the inventory dcs_servers and fill in the new DCS Servers address.

# Force reset the Consul Agent on the target cluster (since HA is in maintenance mode and will not affect the new database cluster)
./nodes.yml -l pg-test -t consul -e dcs_clean=true

When Patroni finishes restarting (in maintenance mode, Patroni restart will not cause Postgres shutdown), it will write the cluster metadata K-V to the new Consul cluster, so you must make sure the Patroni service on the original primary database finishes restarting first.

# Important! Restart Patroni on the target primary first, then restart Patroni on the remaining replica nodes
ansible pg-test-1 -b -a 'sudo systemctl reload patroni'
ansible pg-test-2,pg-test-3 -b -a 'sudo systemctl restart patroni'

45 - FHS: File Hierarchy Structure

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty FHS 

#------------------------------------------------------------------------------
# pigsty
#  ^-----@app                    # extra demo application resources
#  ^-----@bin                    # bin scripts
#  ^-----@docs                   # document (can be docsified)
#  ^-----@files                  # ansible file resources
#            ^-----@conf         # config template files
#            ^-----@rule         # soft link to prometheus rules
#            ^-----@ui           # soft link to grafana dashboards
#  ^-----@roles                  # ansible business logic
#  ^-----@templates              # ansible templates
#  ^-----@vagrant                # sandbox resources
#  ^-----configure               # configure wizard script
#  ^-----ansible.cfg             # default ansible config file
#  ^-----pigsty.yml              # default config file
#  ^-----*.yml                   # ansible playbooks

#------------------------------------------------------------------------------
# /etc/pigsty/
#  ^-----@targets                # file based service discovery targets definition
#  ^-----@dashboards             # static grafana dashboards
#  ^-----@datasources            # static grafana datasources
#  ^-----@playbooks              # extra ansible playbooks
#------------------------------------------------------------------------------

Prometheus FHS 

#------------------------------------------------------------------------------
# Config FHS
#------------------------------------------------------------------------------
# /etc/prometheus/
#  ^-----prometheus.yml              # prometheus main config file
#  ^-----alertmanager.yml            # alertmanger main config file
#  ^-----@bin                        # util scripts: check,reload,status,new
#  ^-----@rules                      # record & alerting rules definition
#            ^-----@infra            # infrastructure rules & alert
#            ^-----@nodes            # nodes rules & alert
#            ^-----@pgsql            # pgsql rules & alert
#            ^-----@redis            # redis rules & alert
#            ^-----@..........       # etc...
#  ^-----@targets                    # file based service discovery targets definition
#            ^-----@infra            # infra static targets definition
#            ^-----@nodes            # nodes static targets definition
#            ^-----@pgsql            # pgsql static targets definition
#            ^-----@redis            # redis static targets definition
#            ^-----@.....            # other targets
#------------------------------------------------------------------------------

Postgres FHS

The following parameters are related to the PostgreSQL database dir:

  • pg_dbsu_home: Postgres default user’s home dir, default is /var/lib/pgsql.
  • pg_bin_dir: Postgres binary dir, defaults to /usr/pgsql/bin/.
  • pg_data: Postgres database dir, default is /pg/data.
  • pg_fs_main: Postgres main data disk mount point, default is /export.
  • pg_fs_bkup: Postgres backup disk mount point, default is /var/backups (optional, you can also choose to backup to a subdirectory on the primary data disk).
#------------------------------------------------------------------------------
# Create Directory
#------------------------------------------------------------------------------
# this assumes that
#   /pg is shortcut for postgres home
#   {{ pg_fs_main }} contains the main data             (MUST ALREADY MOUNTED)
#   {{ pg_fs_bkup }} contains archive and backup data   (MUST ALREADY MOUNTED)
#   cluster-version is the default parent folder for pgdata (e.g pg-test-12)
#------------------------------------------------------------------------------
# default variable:
#     pg_fs_main = /export           fast ssd
#     pg_fs_bkup = /var/backups      cheap hdd
#
#     /pg      -> /export/postgres/pg-test-12
#     /pg/data -> /export/postgres/pg-test-12/data
#------------------------------------------------------------------------------
- name: Create postgresql directories
  tags: pg_dir
  become: yes
  block:
    - name: Make sure main and backup dir exists
      file: path={{ item }} state=directory owner=root mode=0777
      with_items:
        - "{{ pg_fs_main }}"
        - "{{ pg_fs_bkup }}"

    # pg_cluster_dir:    "{{ pg_fs_main }}/postgres/{{ pg_cluster }}-{{ pg_version }}"
    - name: Create postgres directory structure
      file: path={{ item }} state=directory owner={{ pg_dbsu }} group=postgres mode=0700
      with_items:
        - "{{ pg_fs_main }}/postgres"
        - "{{ pg_cluster_dir }}"
        - "{{ pg_cluster_dir }}/bin"
        - "{{ pg_cluster_dir }}/log"
        - "{{ pg_cluster_dir }}/tmp"
        - "{{ pg_cluster_dir }}/conf"
        - "{{ pg_cluster_dir }}/data"
        - "{{ pg_cluster_dir }}/meta"
        - "{{ pg_cluster_dir }}/stat"
        - "{{ pg_cluster_dir }}/change"
        - "{{ pg_backup_dir }}/postgres"
        - "{{ pg_backup_dir }}/arcwal"
        - "{{ pg_backup_dir }}/backup"
        - "{{ pg_backup_dir }}/remote"

PG Binary FHS

On RedHat/CentOS, the default installation location for the Postgres distribution is:

/usr/pgsql-${pg_version}/

The installation playbook automatically creates a soft link to the currently installed version. For example, if version 14 of Postgres is installed, there are.

/usr/pgsql -> /usr/pgsql-14

Therefore, the default pg_bin_dir is /usr/pgsql/bin/, and this path is added to the PATH environment variable for all users in /etc/profile.d/pgsql.sh.

PG Data FHS

Pigsty assumes at least one primary data disk (pg_fs_main) and an optional backup data disk (pg_fs_bkup) on the single node used to deploy the database instance. Usually, the primary data disk is a high-performance SSD, while the backup disk is a high-capacity inexpensive HDD.

#------------------------------------------------------------------------------
# Create Directory
#------------------------------------------------------------------------------
# this assumes that
#   /pg is shortcut for postgres home
#   {{ pg_fs_main }} contains the main data             (MUST ALREADY MOUNTED)
#   {{ pg_fs_bkup }} contains archive and backup data   (MAYBE ALREADY MOUNTED)
#   {{ pg_cluster }}-{{ pg_version }} is the default parent folder
#    for pgdata (e.g pg-test-14)
#------------------------------------------------------------------------------
# default variable:
#     pg_fs_main = /export           fast ssd
#     pg_fs_bkup = /var/backups      cheap hdd
#
#     /pg      -> /export/postgres/pg-test-14
#     /pg/data -> /export/postgres/pg-test-14/data

PG Cluster FHS 

# basic
{{ pg_fs_main }}     /data                      # contains all business data (pg,consul,etc..)
{{ pg_dir_main }}    /data/postgres             # contains postgres main data
{{ pg_cluster_dir }} /data/postgres/pg-test-14  # contains cluster `pg-test` data (of version 13)
                     /data/postgres/pg-test-14/bin            # binary scripts
                     /data/postgres/pg-test-14/log            # misc logs
                     /data/postgres/pg-test-14/tmp            # tmp, sql files, records
                     /data/postgres/pg-test-14/conf           # configurations
                     /data/postgres/pg-test-14/data           # main data directory
                     /data/postgres/pg-test-14/meta           # identity information
                     /data/postgres/pg-test-14/stat           # stats information
                     /data/postgres/pg-test-14/change         # changing records

{{ pg_fs_bkup }}     /var/backups                      # contains all backup data (pg,consul,etc..)
{{ pg_dir_bkup }}    /var/backups/postgres             # contains postgres backup data
{{ pg_backup_dir }}  /var/backups/postgres/pg-test-14  # contains cluster `pg-test` backup (of version 13)
                     /var/backups/postgres/pg-test-14/backup   # base backup
                     /var/backups/postgres/pg-test-14/arcwal   # WAL archive
                     /var/backups/postgres/pg-test-14/remote   # mount NFS/S3 remote resources here

# links
/pg             -> /data/postgres/pg-test-14                 # pg root link
/pg/data        -> /data/postgres/pg-test-14/data            # real data dir
/pg/backup      -> /var/backups/postgres/pg-test-14/backup   # base backup
/pg/arcwal      -> /var/backups/postgres/pg-test-14/arcwal   # WAL archive
/pg/remote      -> /var/backups/postgres/pg-test-14/remote   # mount NFS/S3 remote resources here

Pgbouncer FHS

Pgbouncer is run using the Postgres user, and the config file is located in /etc/pgbouncer. The config file includes.

  • pgbouncer.ini: the main config file
  • userlist.txt: lists the users in the connection pool
  • pgb_hba.conf: lists the access privileges of the connection pool users
  • database.txt: lists the databases in the connection pool

Redis FHS

Pigsty provides essential support for Redis deployment and monitoring.

Redis binaries are installed in /bin/ using RPM-packages or copied binaries, including:

redis-server
redis-server
redis-cli
redis-sentinel
redis-check-rdb
redis-check-aof
redis-benchmark
/usr/libexec/redis-shutdown

For a Redis instance named redis-test-1-6379, the resources associated with it are shown below:

/usr/lib/systemd/system/redis-test-1-6379.service               # Services
/etc/redis/redis-test-1-6379.conf                               # Config
/data/redis/redis-test-1-6379                                   # Database Catalog
/data/redis/redis-test-1-6379/redis-test-1-6379.rdb             # RDB File
/data/redis/redis-test-1-6379/redis-test-1-6379.aof             # AOF file
/var/log/redis/redis-test-1-6379.log                            # Log
/var/run/redis/redis-test-1-6379.pid                            # PID

46 - HA Scenarios

Historical documentation restored from the Pigsty v1.5.1 tag.

You can strengthen your confidence in the cluster’s HA capability through a HA scenario experiment.

Below is a list of 24 typical HA failure scenarios, divided into three categories: Primary failure, Replica failure, and DCS failure, with eight specific scenarios in each category.

All experiments assume that HA auto-switchover mode is enabled, where Patroni should correctly handle Primary and Replica failures.

Number Case Name Auto Mode Manual switch
A Primary Node Failures
1A Primary Node node down Failover Manual switch
2A Primary Node Postgres process shutdown (pg_ctl or kill -9) Failover Manual reboot
3A The Primary Node Patroni process is shut down usually (systemctl stop patroni) Failover Manual reboot
4A Abnormal shutdown of the Primary Node Patroni process (kill -9) Needs confirmation No effect
5A Primary Node load hit full, false death (watchdog) Needs confirmation No effect
6A Primary DCS Agent is not available (systemctl stop consul) Cluster Primary demotion No effect
7A Primary Node network jitter Automatic Failover on timeout Need to observe
8A Erroneous deletion of Primary data dir Automatic Failover Manual switch
B Replica bank failure (1/n , n>1)
1B Replica Node down No effect No effect
2B Replica Node Postgres process shutdown (pg_ctl or kill -9) No effect No effect
3B Replica Node process Postgres Manual Shutdown (pg_ctl) No effect No effect
4B Replica Node Patroni process exception Kill (kill -9) No effect No effect
5B Replica DCS Agent is not available (systemctl stop consul) No effect No effect
6B Replica Node Load hit full, false death Depends Depends
7B Replica Node network jitter No effect No effect
8B Boosting a Replica node by mistake (pg_ctl promte) Automatic recovery Split Brain
C DCS failure
1C DCS Server is completely unavailable (most nodes are unavailable) Downgrade all cluster Primary nodes No effect
2C DCS pass Primary, not Replica (1 Primary & 1 Replica) No effect No effect
3C DCS pass Primary, not Replica (1 Primary n Replica, n>1) No effect No effect
4C DCS pass Replica, not Primary (1 Primary, 1 Replica) No effect No effect
5C DCS pass Replica, not Primary (1 Primary n Replica, n>1) Automatic Failover No effect
6C DCS network jitter: simultaneous outages,
Primary and Replica nodes recover simultaneously, or the Primary node recovers first		 No effect No effect
7C DCS network jitter: simultaneous outages,
Replica nodes recover first, Primary nodes recover later (1 Primary, 1 Replica)		 No effect No effect
8C DCS network jitter: simultaneous interruptions,
Replica nodes recover first, Primary nodes recover later (1 Primary n Replica, n>1)		 Automatic Failover over TTL No effect

Environment

The following is a walkthrough of a local Pigsty four-node sandbox.

Prepare load

In the experiment, you can use pgbench to generate virtual loads and observe the state of load traffic under various failures.

make test-ri     # Initialize the pgbench table in the pg-test cluster
make test-rw     # Generate pgbench write traffic
make test-ro     # Generate pgbench read-only traffic

If you wish, you can directly adjust the load generation commands and execute them if you want to emulate other traffic styles.

# 4 connections, total 64 read/write TPS
while true; do pgbench -nv -P1 -c4 --rate=64 -T10 postgres://test:test@pg-test:5433/test; done

# 8 connections, total 512 read-only TPS
while true; do pgbench -nv -P1 -c8 --select-only --rate=512 -T10 postgres://test:test@pg-test:5434/test; done

Observation Status

The PGSQL Cluster panel provides important monitor information about the pg-test cluster. You can review the last 5-15 minutes of metrics and automatically set it to refresh every 5 seconds.

Pigsty’s monitor metrics collection period is 10 seconds by default, while the typical time taken for Patroni Primary-Replica switchover is usually between a few and a dozen seconds. You can use patronictl to obtain sub-second observation accuracy.

pg list pg-test          # View pg-test cluster status (in a separate window)
pg list pg-test -w 0.1   # View pg-test cluster status, refreshed every 0.1s

You can open four Terminal windows for.

  • Execute admin commands on the meta node (the command used to trigger a simulated failure).
  • Initiate and observe read and write request loads (pgbench).
  • Initiate and observe read-only request load (pgbench --select-only).
  • Real-time access to cluster Primary-Replica status (pg list).

Primary Failure

1A-Primary Node Down

Operating Instructions

ssh 10.10.10.3 sudo reboot    # Reboot the pg-test-1 Primary node directly (VIP points to the actual Primary node)

Operation results

Patroni can handle the Primary node downtime, typically performing automatic Failover.

Manual intervention is required when the cluster is in maintenance mode (manual execution of pg failover <cluster>).

patronictl list results 
# Normal: pg-test-3 is the current cluster Primary with a timeline of 3 (this cluster has experienced two Failovers)
+ Cluster: pg-test (7037005266924312648) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Replica | running |  3 |         0 | clonefrom: true |
| pg-test-2 | 10.10.10.12 | Replica | running |  3 |         0 | clonefrom: true |
| pg-test-3 | 10.10.10.13 | Leader  | running |  3 |           | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+

# ssh 10.10.10.13 sudo reboot reboots the node where the pg-test-3 Primary instance resides and the patroni of the pg-test-3 instance disappears from the cluster
# After going offline for more than TTL, the pg-test-1 instance grabs the Leader Key and becomes the new cluster leader.
+ Cluster: pg-test (7037005266924312648) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  3 |           | clonefrom: true |
| pg-test-2 | 10.10.10.12 | Replica | running |  3 |         0 | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+

# The pg-test-1 instance completes the Promote and becomes the new leader of the cluster, and the timeline changes to 4
+ Cluster: pg-test (7037005266924312648) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  4 |           | clonefrom: true |
| pg-test-2 | 10.10.10.12 | Replica | running |  3 |         0 | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+

# The pg-test-2 instance modifies its upstream to the new leader, pg-test-1, and the timeline changes from 3 to 4, entering a new era and looking at the new core.
+ Cluster: pg-test (7037005266924312648) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  4 |           | clonefrom: true |
| pg-test-2 | 10.10.10.12 | Replica | running |  4 |         0 | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+

# pg-test-3 finished restarting, Postgres is in a stopped state, Patroni rejoins the cluster
+ Cluster: pg-test (7037005266924312648) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  4 |           | clonefrom: true |
| pg-test-2 | 10.10.10.12 | Replica | running |  4 |         1 | clonefrom: true |
| pg-test-3 | 10.10.10.13 | Replica | stopped |    |   unknown | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+

# Postgres on pg-test-3 is pulled up as a Replica to synchronize data from the new Primary, pg-test-1.
+ Cluster: pg-test (7037005266924312648) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  4 |           | clonefrom: true |
| pg-test-2 | 10.10.10.12 | Replica | running |  4 |         1 | clonefrom: true |
| pg-test-3 | 10.10.10.13 | Replica | running |  3 |        10 | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+

# pg-test-3 Catch up with the new leader, timeline goes to 4, keep up with the new leader.
+ Cluster: pg-test (7037005266924312648) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  4 |           | clonefrom: true |
| pg-test-2 | 10.10.10.12 | Replica | running |  4 |         1 | clonefrom: true |
| pg-test-3 | 10.10.10.13 | Replica | running |  4 |         0 | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+

2A-Primary Postgres Process Shutdown

Operating Instructions

Two different ways to shut down the Primary Postgres instance: regular pg_ctl and brute-force kill -9.

# Shut down the Postgres Primary process on the mater node
ssh 10.10.10.3 'sudo -iu postgres /usr/pgsql/bin/pg_ctl -D /pg/data stop'

# Query the Primary PID and force a Kill
ssh 10.10.10.3 'sudo kill -9 $(sudo cat /pg/data/postPrimary.pid | head -n1)'

Operation results

After shutting down Postgres, Patroni tries to pull up the Postgres process again. If successful, the cluster returns to normal.

If the PostgreSQL process cannot be pulled up correctly, the cluster will automatically Failover.

patronictl list results 
# After the Primary node instance is forced to kill, the status is shown as crashed, and then immediately pulled back up and restored to Running
+ Cluster: pg-test (7037005266924312648) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Leader  | crashed |    |           | clonefrom: true |
| pg-test-2 | 10.10.10.12 | Replica | running |  7 |         0 | clonefrom: true |
| pg-test-3 | 10.10.10.13 | Replica | running |  7 |         0 | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+

# If the persistent Kill Primary causes the Primary to fail to pull up (status changes to start failed), then Failover will be triggered
+ Cluster: pg-test (7037005266924312648) ----------+----+-----------+-----------------+
| Member    | Host        | Role    | State        | TL | Lag in MB | Tags            |
+-----------+-------------+---------+--------------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Replica | running      | 11 |         0 | clonefrom: true |
| pg-test-2 | 10.10.10.12 | Leader  | running      | 12 |           | clonefrom: true |
| pg-test-3 | 10.10.10.13 | Replica | start failed |    |   unknown | clonefrom: true |
+-----------+-------------+---------+--------------+----+-----------+-----------------+

3A-Primary Patroni Process Shutdown

Operating Instructions

# Shut down the Postgres Primary process on the Primary node
ssh 10.10.10.3 'sudo systemctl stop patroni'

Operation results

Shutting down the Primary Patroni commonly causes the PostgreSQL instances managed by Patroni to shut down together and immediately trigger a cluster Failover.

Shutting down Patroni in maintenance mode normally, shutting down Patroni does not affect the managed PostgreSQL instances, which can restart Patroni to reload the config (e.g., change the DCS used).

patronictl list results 
# Primary node Patroni (pg-test-3) after shutdown
+ Cluster: pg-test (7037370797549923387) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Replica | running |  2 |         0 | clonefrom: true |
| pg-test-2 | 10.10.10.12 | Replica | running |  2 |         0 | clonefrom: true |
| pg-test-3 | 10.10.10.13 | Leader  | running |  2 |           | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+

# pg-test-3 enters the stopped state
+ Cluster: pg-test (7037370797549923387) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Replica | running |  2 |         0 | clonefrom: true |
| pg-test-2 | 10.10.10.12 | Replica | running |  2 |         0 | clonefrom: true |
| pg-test-3 | 10.10.10.13 | Replica | stopped |    |   unknown | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+

# New Primary node pg-test-2 When selected, the timeline goes from 2 to 3
+ Cluster: pg-test (7037370797549923387) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Replica | running |  2 |         0 | clonefrom: true |
| pg-test-2 | 10.10.10.12 | Leader  | running |  3 |           | clonefrom: true |
| pg-test-3 | 10.10.10.13 | Replica | stopped |    |   unknown | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+

# Another healthy Replica, pg-test-1, re-follows the new Primary, pg-test-2, into timeline 3, and the old Primary, pg-test-3, disappears from the cluster after some time
+ Cluster: pg-test (7037370797549923387) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Replica | running |  3 |         0 | clonefrom: true |
| pg-test-2 | 10.10.10.12 | Leader  | running |  3 |           | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+

# Use systemctl start patroni to pull up the old Primary node pg-test-3 again, and the instance automatically enters replication mode, following the new leader.
+ Cluster: pg-test (7037370797549923387) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Replica | running |  3 |         0 | clonefrom: true |
| pg-test-2 | 10.10.10.12 | Leader  | running |  3 |           | clonefrom: true |
| pg-test-3 | 10.10.10.13 | Replica | running |  3 |         0 | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+

4A-Primary Node Patroni process abnormally shut down

Warning

This situation requires special attention!

Suppose you use Kill -9 to kill the Primary Patroni forcibly. In that case, there is a high probability that the Primary Patroni will not be able to shut down the managed PostgreSQL Primary instances. This will cause the original Primary PostgreSQL instance to survive Patroni’s death. At the same time, the remaining Replica nodes in the cluster will hold a leadership election to elect a new Primary, leading to a split-brain.

Operating Instructions

# Shutting down the Patroni Primary process on the Primary node
ssh 10.10.10.3 "ps aux | grep /usr/bin/patroni | grep -v grep | awk '{print $2}'"
ssh 10.10.10.3 'sudo kill -9 723'

Operation results

This operation may cause a cluster split-brain: because Patroni dies violently and cannot kill the PostgreSQL process it manages. Instead, the other cluster members elect a new Primary node after the TTL timeout.

If you use the standard load balancing health check-based service access mechanism, there will be no problem because the original Primary node Patroni is dead, and the health check is false. The LB will not distribute traffic to this instance even if that Primary is alive. However, if you continue to write to this Primary by other means, you may have a split-brain!

Patroni uses the Watchdog mechanism to underwrite this situation, which you need to use as appropriate (parameter patroni_watchdog_mode). When watchdog is enabled, if the original Primary cannot shut down the PG Primary in time to avoid split-brain in Failover for various reasons (Patroni crash, machine load fake death, VM scheduling, PG shutdown too slow), etc., the Linux kernel module softdog will be used to force a shutdown to avoid split-brain.

patronictl list results 
# Use Kill-9 to force kill the primary Patroni (pg-test-2)
+ Cluster: pg-test (7037370797549923387) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Replica | running |  3 |         0 | clonefrom: true |
| pg-test-2 | 10.10.10.12 | Leader  | running |  3 |           | clonefrom: true |
| pg-test-3 | 10.10.10.13 | Replica | running |  3 |         0 | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+
# Because Patroni dies, the PostgreSQL process is usually still alive and in Primary state
# Because Patroni died, the health check of the original Primary node will immediately fail, resulting in no instances of Primary traffic being carried and the cluster being unwritable.

# Because Patroni dies violently, it has no time to release the Leader Key in the DCS, so the above state will remain TTL for a long time.
# It is not until the Leader Lease in DCS is released due to timeout (about 15s) that the cluster realizes the Primary is dead and initiates a Failover
+ Cluster: pg-test (7037370797549923387) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Replica | running |  3 |         0 | clonefrom: true |
| pg-test-3 | 10.10.10.13 | Replica | running |  3 |         0 | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+

# The cluster triggers Failover, pg-test-1 becomes the new cluster leader and starts carrying read-only traffic, and the cluster write service resumes
+ Cluster: pg-test (7037370797549923387) -----+----+-----------+-----------------+
| Member    | Host        | Role    | State   | TL | Lag in MB | Tags            |
+-----------+-------------+---------+---------+----+-----------+-----------------+
| pg-test-1 | 10.10.10.11 | Leader  | running |  4 |           | clonefrom: true |
| pg-test-3 | 10.10.10.13 | Replica | running |  3 |         0 | clonefrom: true |
+-----------+-------------+---------+---------+----+-----------+-----------------+

# Attention must be paid at this point! The original cluster Primary is still alive and allowing writes!!!
# If you use the standard load balancing health check based traffic distribution mechanism, there will be no problem because Patroni is dead and the health check is false.
# The Primary is alive, but the LB will not distribute traffic to this instance. However, if you write directly to this Primary through other means, you will have a split-brain!
$ psql -AXtwh 10.10.10.12 -d postgres -c 'select pg_is_in_recovery();'
t

Recovering from this situation

When Patroni dies violently, first manually shut down the original PostgreSQL Primary that is managed by it and still running, then restarts Patroni again and have the PostgreSQL instance pulled up by Patroni.

/usr/pgsql/bin/pg_ctl -D /pg/data stop
systemctl restart patroni

If not, an error may occur where Patroni does not start properly.

2021-12-03 14:16:18 +0800 INFO:  stderr=2021-12-03 14:16:18.752 HKT [7852] FATAL:  lock file "postPrimary.pid" already exists
2021-12-03 14:16:18.752 HKT [7852] HINT:  Is another postPrimary (PID 887) running in data directory "/pg/data"?

Explanation of parameter patroni_watchdog_mode.

  • If mode is required but /dev/watchdog is not available, it will not affect Patroni startup, only the leadership candidacy of the current instance.
  • If the mode is required, but /dev/watchdog is not available. The instance cannot be a qualified Primary candidate, i.e., it cannot participate in Failover, even if manually forced to specify it: a Switchover failed, details: 412, switchover is not possible: no good candidates have been found error. To solve this problem, change the patroni_watchdog option in the /pg/bin/patroni.yml file to automatic|off.
  • If the mode is automatic, there is no restriction, and the instance will be able to run in the Primary election regardless of whether /dev/watchdog is available or not.
  • Two conditions are required for /dev/watchdog to be available, the softdog kernel module is loaded, and /dev/watchdog is owned by postgres (dbsu).

5A-Primary DCS Agent is not available

In this case, the Patroni on the Primary will demote itself to a normal Replica because it cannot connect to the DCS service. However, if the Replica Patroni is still aware that the Primary is alive (e.g., streaming replication is still going on normally), it does not trigger Failover!

In this case, Pigsty’s access mechanism will cause the whole cluster to enter a Primaryless state and be unwritable because the original Primary node health check is false!

In maintenance mode, no changes.

6A-Primary Node load hit full, false death

The v1.5.1 source reserved this failure scenario but did not provide an additional procedure.

7A-Primary Node network jitter

8A-Made-deletion of the Primary data dir

Replica node failure experiment

1B-Replica node down

Operating Instructions

ssh 10.10.10.3 sudo reboot    # Reboot the pg-test-1 Primary node directly (VIP points to the actual Primary node)

Operation Result

A Replica node going down will cause services such as HAPorxy, Patroni, Postgres, etc., on that node to become unavailable. Usually, the business side will notice a minimal number of transient error reports (the connection to the failed instance will be broken), and then the other LBs in the cluster will take this failed node off the backend list.

Note that if the cluster is one Primary & one Replica and the only Replica is down, the offline query service may be affected (no available bearer instances).

Once the node restart is complete, the Patroni service will automatically pull up, and the instance will automatically rejoin the cluster.

2B-Replica node Postgres process shutdown

Operating Instructions

Two different ways of shutting down the Replica Postgres instance: regular pg_ctl and brute-force kill -9.

# Shut down the Postgres Primary process on the Replica node
ssh 10.10.10.3 'sudo -iu postgres /usr/pgsql/bin/pg_ctl -D /pg/data stop'

# Query the Replica PID and force Kill
ssh 10.10.10.3 'sudo kill -9 $(sudo cat /pg/data/postPrimary.pid | head -n1)'

Operation Results

After shutting down Postgres, Patroni tries to pull up the Postgres process again. If successful, the cluster returns to normal. If the Replica goes down causes the health check for that instance to be Down, the cluster’s LB will not redistribute traffic to that instance, and a few transient errors will be reported for application read-only requests.

3B-Replica node Postgres process manually shut down

4B-Replica Patroni process abnormally kills

5B-Replica DCS Agent is not available

6B-Replica node load hit full, fake death

7B-Replica network jitter

8B-Mislocated a Replica node

DCS Failure Experiment

1C-DCS Server is completely unavailable

DCS is completely unavailable is a severe failure that will cause all database clusters to be unwritable by default. If L2 VIP access is used, the L2 VIP bound to the Primary node is also unavailable by default, resulting in the entire cluster being potentially unreadable!

DCS can solve this problem: by using a distributed architecture with a reliable disaster recovery mechanism. For example, a 3-node DCS cluster allows one server to fail, while a 5-node DCS cluster allows up to two server nodes to fall simultaneously.

After shutting down Consul, all database cluster Primary with HA auto-switchover mode enabled to trigger the demotion logic (because the Patroni of the Primary are not aware of the presence of other cluster members and have to assume that the other Replicas already constitute a quorum majority of the partition and are elected, thus demoting themselves as Replicas to avoid split-brain).

Operating Instructions

Shut down the DCS Server on the meta node, at least two if three and at least three if there are 5.

systemctl stop consul

Solution

  1. In maintenance mode, the user loses the ability to automatically Failover, but a DCS failure will not cause the Primary node to be unwritable. (fast manual switchover is still possible).
  2. Use more DCS instances to ensure DCS availability (DCS itself was created to solve this problem).
  3. Configure a long enough timeout retry time for Patroni and set the highest response priority for DCS failures.

2C-DCS pass Primary, not a Replica (1 Primary, 1 Replica)

3C-DCS pass Primary, no Replica (1 Primary n Replica, n>1)

4C-DCS pass Replica, not pass Primary (1 Primary, 1 Replica)

5C-DCS pass Replica, no Primary (1 Primary n Replica, n>1)

6C-DCS network jitter: simultaneous outages, Primary and Replica recover at the same time, or Primary recovers first

7C-DCS network jitter: simultaneous interruption, Replica recovers first, Primary recovers later (1 Primary, 1 Replica)

8C-DCS network jitter: simultaneous interruption, Replica recovers first, Primary recovers later (1 Primary n Replica, n>1)

47 - Common Failures

Historical documentation restored from the Pigsty v1.5.1 tag.

Hardware Faults

ID Name Symptom Process
H01 Primary node down pg_up = 0 for 1-3 minutes No immediate intervention required.
Additional examples after the fact
Removal from Access Domain
Execute Case 8: Cluster Role Adjustment
H02 Replica node down pg_up = 0 for 1-3 minutes No immediate intervention required
Adding examples after the fact.
Removal from Access Domain
Execute Case 8: Cluster Role Adjustment
H03 Primary node network partition Loss of all monitoring data of the primary instance, network unreachable Confirm Failover status
Force Fencing old primary if necessary
H04 Replica Node Network Partitioning Loss of all monitoring data from the instance, network unreachable Usually no effect, waiting for recovery
Contact O&M and network engineers to handle
H05 TCP the retransmission rate is too high TCP Retrans stay high for a long time, a lot of Conn Reset, a lot of query requests fail Find O&M and network engineers to handle
H06 Node memory error EDAC counter growth, system error log After confirming that there are no errors in the replica memory
Execute Case 10: cluster primary-replica switch
H07 Bad blocks on disk, data corruption Query results, and logs show error messages such as can’t read block Execute Case 10: cluster primary-replica switch
Manual data recovery using data recovery tools
R01 High CPU usage CPU / load / pressure index high topCheck for large CPU footprint programs and clean them up
As in the case of an avalanche, execute a kill query stop.
R02 OOM appears Process Failure appears, OOM message, high memory usage, start using SWAP Confirm memory, confirm SWAP
topCheck for large memory hogs and clean them up
Re-pulling the killed process
Emergency SWAP partition addition
R03 Disk Full Disk Write Full
Database Crash
A large number of shell commands cannot be executed		 Remove /pg/dummy to free up emergency space
Check and handle WAL buildup
Check aa and process a large number of Log files
Confirm whether the business has cleanable data
R06 Disk/network card IO too high Disk/NIC BandWidth too large
Disk > 2GB/s
Network > 1 GB/s		 Check applications that use the network/disk, such as backups, to add speed limits.

Software Errors

ID Name Symptom Process
SP1 Database process abort ps aux can’t find the postgres process Check Postgres, Patroni status
Confirm Failover results, or perform Failover manually
SP2 Connection pool process aborted systemctl status pgbouncer Failure restart service component or reset service component
SP3 Primary Patroni process aborted systemctl status patroni Failure As above, enter maintenance mode, reboot or reset Patroni
SP4 Primary Consul process aborted systemctl status consul Failure As above, enter maintenance mode, reboot or reset Consul
S05 HAProxy process aborts systemctl status haproxy Failure As above, restart or reset Haproxy
S06 Connection pool contamination An error message similar to Cannot execute XXX on read-only transactions appears Restart the Pgbouncer connection pool
or configure server_reset_query
S07 Connection pool cannot connect to the database pgbouncer can not connect to server Check whether the user, password, and HBA configuration are correct
Execute Case-4: Cluster Service User Creation to refresh the user
S08 Connection pool reaches QPS bottleneck PGbouncer QPS reaches 3 to 4W, CPU usage reaches 100% Use multiple Pgbouncers (not recommended)
Use Default service to bypass Pgbouncer
Notify business side of speed limit
S09 DCS Server is not available In auto-switchover mode, all primary will go to the unwritable state after TTL Set all clusters to maintenance mode immediately
S10 DCS Agavailableavailable If it is a replica, it has no effect, if it is a primary, it will be demoted to a replica and the cluster is not writable Set all clusters to maintenance mode immediately
S11 XID Wraparound Enter protection mode when age remaining 1000w. This problem should be avoided in advance through monitoring
locate the over-aged databases and tables, perform emergency cleaning
quickly locate the cause of blocking the vacuum and solve
restore in single user mode
S12 WAL Stacking WAL size continues to grow Execute CHECKPOINTmultiple times
confirm the wal archive status
confirm whether there are unfinished ultra-longg transactions from the replica
confirm whether there are replication slots to prevent wal recycling

Human Errors

ID Name Symptom Process
M01 Mistakenly deleted database clusters The database cluster is gone Use cold standby to recover the cluster
Prepare to run
M02 Mistakenly elevating an instance to the primary split-brain No need to handle it in automatic mode, otherwise, split-brain
M03 Erased data by mistake The data is gone Stop vacuum, use PG_ Dirtyread extract
extract from delayed cluster
extract and restore from cold standby
M04 Erasure Form The table is gone Fetch from delayed cluster
Fetch and restore from cold standby
M05 Integer Sequence Number Overflow Sequence exceeds INTMAX Refer to integer primary key online upgrade manual to handle
M06 Insert data conflicts due to duplicate primary key serial numbers violate constratint … Grow serial number value (e.g. +100000)
M07 Slow query queuing / avalanche Large number of slow query logs Use pg_terminate_backend to periodically clean up slow queries (e.g. every 1 second)
M08 Deadlock queuing / avalanche Lock stacking Use pg_terminate_backend to periodically clean up queries (e.g. every 1 second)
M09 HBA denied access no HBA entry for xxx Case 6: APPLY-PGSQL-HBA
M10 User password error password auth failure for xxx Case 4: Create OGSQL Biz User
M11 Insufficient access privileges permission denied for x Check if the user created the object with the correct admin
Refer to Default Privilege to manually fix the object privileges

48 - Community

Historical documentation restored from the Pigsty v1.5.1 tag.

There’s an active Pigsty community. Search WeChat pigsty-cc and join the user group.

Most answers cloud be found in FAQ. If there’s no answer to your questions, you can ask in the community.

It’s most welcome to create Issues or Pull Requests if you find some repeatable bugs.

Following information cloud helps a lot when asking for help:

  • What commands are you typing?
  • What result do you expect?
  • What environment is used?
    • Is it a CentOS 7.8 fresh node?
    • Is it a physical/virtual machine, or a public cloud ECS?
    • Is anything else running on that node?
    • Is offline packagepkg.tgz been used?
      • If yes, in /tmp/pkg.tgz, right?
      • If not, do you have proper Internet access?
    • Is there any special restriction ? (such as firewall, ssh ban, etc…)

Mail

Github Issues

Telegram

Discord

https://discord.gg/wDzt5VyWEz

49 - Roadmap

Historical documentation restored from the Pigsty v1.5.1 tag.

History

Time Description Release
2019-05-15 POC fa2ade31 (v0.0.1)
2020-04-30 First Commit dd64677 (v0.0.2)
2020-06-20 Validation on Testing Environment v0.1.0
2020-06-22 Interface enhancement v0.0.3
2020-07-10 PGSQL Monitoring v6 GA v0.2.0
2020-07-27 Refactor playbooks into ansible roles v0.0.4
2020-08-19 Offline Installation Mode v0.0.5
2020-10-22 Provisioning Solution GA v0.3.0
2020-12-14 PostgreSQL 13 Support, Official Documentation v0.4.0
2021-01-07 Databaes Customize Template v0.5.0
2021-02-19 Architecture Enhancement v0.6.0
2021-03-01 Monitor only deployment v0.7.0
2021-03-28 Service Provision v0.8.0
2021-04-04 Pigsty GUI, CLI, Logging Intergration v0.9.0
2021-04-20 Accessibility and extensibility enhancement v0.9.1
2021-07-26 v1 GA, Monitoring System Overhaul v1.0.0
2021-10-12 HomePage, JupyterLab, Pgweb, Pev2 & Pgbadger v1.1.0
2021-11-03 Upgrade default Postgres to 14, monitoring existing pg v1.2.0
2021-11-30 PGCAT Overhaul & PGSQL Enhancement & Redis Support Beta v1.3.0
2022-03-31 MatrixDB Support, Separated INFRA, NODES, PGSQL, REDIS v1.4.0
2022-04-20 Bug fix & Full translation of English documents. v1.4.1
2022-05-31 Docker Applications v1.5.0

Timeline

50 - Development Log

Historical documentation restored from the Pigsty v1.5.1 tag.

2022-06-18

  • 1.5.1 for PostgreSQL 14.4
  • Upgrade haproxy to 2.6.0
  • repotrack i686 issus on el7
  • New apps: wiki.js ferretDB

2022-05-21

  • Grafana Logo Customize
  • Nginx Log Rework

2022-05-20

  • Prometheus Overview polish
  • Grafana Overview polish
  • DCS Overview polish

2022-05-15

  • new role: etcd
  • monitoring for etcd
  • Make etcd a viable option for ha PG DCS
  • Infra Dashboards rework
  • fix nodes/infra/pgsql config docs
  • Update docker containers, pgadmin4 to 6.9

2022-05-14

  • fix redis join issue
  • optimize consul role implementation

2022-05-13

  • pg_exporter metrics & dashboard calibration

2022-05-12

  • upgrade bytebase to v1.0.5

2022-05-09

  • v1.5.0-alpha.1 released
  • repo upstream calibration
  • build docker.tgz image cache and add to download script

2022-05-07

  • grant backup related permissions to pg_replication_username
  • add pg_probackup to default packages

2022-05-06

  • Docker application calibration

2022-05-05

  • New CMDB Design
  • Now cmdb works for redis & gpsql

2022-05-04

  • Dashboard Overhaul: Infra-Overview
  • Add node_crontab_overwrite implementation
  • Test citus/dcs3/mxdb conf
  • node_local_repo_url -> node_repo_local_urls

2022-05-02

  • add crontab implementation

2022-05-01

  • Refactor redis.yml & redis-remove.yml
  • Remove role redis, redis-remove, redis-exporter
  • Fix dashboard redis aliveness issue
  • Upgrade to redis v6.2.7

2022-04-30

  • Register Infra as common consul service
  • Make consul service discovery great again
  • implement node_data_dir which create /data during node init if not exists
  • add pg_probackup to default package list
  • integration test on pigsty-citus

2022-04-29

New Vars:

node_data_dir node_crontab

Bulk Rename:

node_ntp_config -> node_ntp_enabled node_admin_setup -> node_admin_enabled node_admin_pks -> node_admin_pk_list node_dns_hosts -> node_etc_hosts_default node_dns_hosts_extra -> node_etc_hosts node_dns_server -> node_dns_method node_local_repo_url -> node_repo_local_urls node_packages -> node_packages_default node_extra_packages -> node_packages node_packages_meta -> node_packages_meta node_meta_pip_install -> node_packages_meta_pip node_sysctl_params -> node_tune_params dcs_name -> dcs_name dcs_exists_action -> dcs_clean dcs_disable_purge -> dcs_safeguard app_list -> nginx_indexes grafana_plugin -> grafana_plugin_method grafana_cache -> grafana_plugin_cache grafana_plugins -> grafana_plugin_list grafana_git_plugin_git -> grafana_plugin_git haproxy_admin_auth_enabled -> haproxy_auth_enabled pg_exists_action -> pg_clean pg_disable_purge -> pg_safeguard pg_shared_libraries -> pg_libs

2022-04-28

  • Upgrade Loki to v2.5.0 with new rpm packager nfpm
  • Upgrade pg_exporter to v0.5.0 with new rpm packager nfpm
  • Upgrade grafana version to 8.4.6
  • Upgrade Consul to 1.13
  • Upgrade Pgbouncer to 1.17
  • Upgrade vip-manager to 1.0.2
  • Upgrade Grafana to 8.5.0
  • Remove pgweb by default (since docker will cover it)
  • Remove role jupyter, move it’s logic into infra-jupyter.yml
  • Refactor docker role, it can be enabled per nodes.

2022-04-26

Add new options:

nameserver_enabled: false  # setup dnsmasq
prometheus_enabled: true   # setup prometheus
grafana_enabled: true      # setup grafana
docker_enable: true        # setup docker
loki_enabled: true         # setup loki

2022-04-24

English Document Released!

2022-03-30

  • Pigsty v1.4.0 Released!

2022-03-24

  • Get the latest source with curl -SL https://github.com/Vonng/pigsty/releases/download/v1.5.1/pigsty.tgz | gzip -d | tar -xC ~
  • add download script to get pigsty/pkg/app/matrix packages from github
  • if Github is not viable, use CDN instead.

2022-03-20

  • Now playbooks are divided into 4 major groups: infra, nodes, pgsql, redis
  • add promtail to infra.yml, nodes.yml, etc…

2022-03-19

  • build haproxy , loki, promtail rpm for pigsty
  • enable loki by default
  • enable promtail by default
  • promtail now collect nodes

2022-03-08

  • Greenplum/MatrixDB Support finished!
    • Simplify pigsty-matrixdb.yml playbook
    • Remove role gp_prepare & gp_provision
    • Remove playbook gpsql-post.yml
    • Update default pigsty-mxdb.yml example
  • Fix all example configs
  • Add -d postgres when execute pgsql-createuser.yml
  • Software update:
    • pg_exporter v0.4.1 : new timeout parameter, bug fix, pgbouncer v1.16 support, etc…
    • greenplum 6.19.1 -> 6.19.3
    • grafana 8.4.2 -> 8.4.3

2022-03-05

  • Overhaul dashboards
    • Nodes Cluster / Nodes Instance / Nodes Overview
    • Add ds datasource variable to all dashboards (so you can change prometheus datasource by select hidden variables)
    • Reforge PGSQL Cluster Dashboards
    • Add new dashboards: PGSQL Databases which focus on database dimension among cluster
    • Adjust links for pgsql-node to nodes-instance
    • Add database & service nav for pgsql-overview

2022-02-22

  • software upgrade

    • postgres minor version update: 14.1 -> 14.2
    • prometheus version 2.33
    • timescaledb 2.6.0
    • postgis 3.2
    • grafana 8.4.2

  • add ins , cls label to node metrics

2022-02-14

  • PGSQL Overview Dashboard Rework
  • PGSQL Cluster Dashboard Reforge

2022-02-13

  • Node Overview Dashboard
  • Node Instance Dashboard

2022-02-02

  • Add matrixdb support

2022-01-28

  • Dashboard adjustment for new label structure

2022-01-25

  • Redesign ip based node metrics
  • refactor on pg_exporter node_exporter deployment

2022-01-24

  • Software upgrade
    • HAProxy 2.2 -> 2.5
    • Greenplum: 6.18 -> 6.19
    • Pev2 : 0.23 -> 0.24
    • pgweb : 0.11.9 -> 0.11.10
    • loki, promtail, logcli, loki-canary: 2.4.1 -> 2.4.2

2022-01-22

  • Monitoring v9 Launch
    • split nodes monitoring from infra & pgsql
    • re-forge on labels, keys & etc..
  • Reforge prometheus rules: infra, nodes, pgsql, redis, …

2022-01-21

  • split role monitor into pg_exporter & node_exporter
  • add new parameter: pg_exporter_params
  • upgrade default haproxy version from 2.2 to 2.5
  • bug fix: pip3 install jupyter failed now a pip3 upgrade is performed before download & install

2021-12-29

  • disable repo on other meta nodes in infra-demo.yml Now bootstrap on multiple meta nodes are much easier
  • enhancement: remove Require=consul from patroni systemd service Which makes dcs migration much easier
  • fix pg_exporter.yml pg_index column sequence Which is a workaround for a known bug of PostgreSQL

2021-12-17

  • Add terraform support for Aliyun

2021-12-09

  • Redis Dashboards Enhancement
  • PGCAT Dashboards Enhancement
  • Separate loki & pgweb from standard infra.yml playbook

2021-12-04

  • v1.3.1 Release
  • Bug fix
    • configure check_bin software version
    • add citus to pigsty-pub4 pg-meta cluster by default
    • add auth parts to environ patronictl.yml
  • add option citus.node_conninfo: ‘sslmode=prefer’ for all conf template
  • add example configuration files: citus, dcs3, pub4
  • pg alias for patronictl on meta nodes
  • add patroni to prometheus targets (since 2.1.1)

2021-12-03

  • add ftp to package list
  • add one-pass init playbook (for 3-dcs x 3-node deploy)
  • add reloadha & reloadhba shell shortcuts

2021-11-29

  • Add redis support for pigsty
  • pgcat overhaul
  • new key metrics panel for pgsql cluster & pgsql instance ….

2021-09-23

  • [ENHANCEMENT] home page overhaul
  • [ENHANCEMENT] add jupyter lab integration
  • [ENHANCEMENT] add pgweb console integration
  • [ENHANCEMENT] update default pkg.tgz software version:

2021-09-18

  • add pev2 support
  • add pgbadger support

2021-09-17

  • [ENHANCEMENT] add pg_dummy_filesize to create fs space placeholder

2021-09-14

  • release v1.0.1
  • huge amount of documentation updates
  • fix some minor bugs

2021-09-14

  • Documentation Update
    • Chinese document now viable
    • Machine-Translated English document now viable
  • Bug Fix: pgsql-remove does not remove primary instance.
  • Bug Fix: replace pg_instance with pg_cluster + pg_seq
    • Start-At-Task may fail due to pg_instance undefined
  • Bug Fix: remove citus from default shared preload library
    • citus will force max_prepared_transaction to non-zero value
  • Bug Fix: ssh sudo checking in configure:
    • now ssh -t sudo -n ls is used for privilege checking
  • Typo Fix: pg-backup script typo
  • Alert Adjust: Remove ntp sanity check alert (dupe with ClockSkew)
  • Exporter Adjust: remove collector.systemd to reduce overhead

2021-07-22

  • refactor load_config.sh with python to load_conf.py
  • add inventory_cmdb and inventory_conf to switch between static config and dynamic inventory
  • add new dashboards: pgsql-activity which focus on cluster level activities
  • add new dashboards: pgcat-bloat which focus on table & index bloat
  • fix minor dashboard bugs
  • fix pg_exporter config pg_repl collector version overlap for PG12
  • update covid dashboards & isd dashboards
  • prepare for 1.0.0

2021-07-21

  • add ipython jupyterlab to meta packages
  • add pip support for meta python env
  • add grafana plugins echarts & csv/json data source
  • prepare for v1.0.0-beta2

2021-07-20

  • bug fix: register role does not run on all meta nodes
  • add check for createpg createdb createuser scripts

2021-07-19

  • Use docsify as documentation solution

2021-07-15

  • Calibration of dashboard data links
  • remove grafana & prometheus database definition in config file instead, change grafana primary database to postgres will be a tutorial for get start with pigsty
  • build & fix dashboard data links
  • add catlog pglog alias for meta node environment (get and pour pgsql log)
  • change default timezone from Asia/ShangHai to Asia/Hong_kong
  • add pg_shared_libraries to customize extensions
  • install citus, timescaledb by default
  • prepare for v1.0.0-beta1

2021-07-14

  • add pgsql-xacts dashboard
  • add pgsql-persist dashboard
  • add pgsql-tables dashboard
  • add pglog-analysis dashboard
  • add pglog-session dashboard
  • add softlinks in files dir to dashboards

2021-07-13

  • Release pigsty v1.0.0-alpha2
  • Fix systemd-devel deps failure on VPC.
  • remove core apps
  • refactor cmdb application, use pg_datbases.meta.baseline to provisioning cmdb schema
  • add load_config to parse and activate config from config file
  • Integrate pglog schema into pigsty schema
  • Update files/conf

2021-07-12

  • alert panel now links to alertmanager
  • add click-able data-link to most graph
  • release pg_exporter v0.4.0 (remove beta)
  • adjust home & overview & cluster & instance layout
v1.0.0-alpha2 Release

* New Dashboard:  `pgcat-query`, get detailed information (including raw sql string) directly from corresponding postgres datasource
* New Dashboard:  `pgsql-queries`, runs on instance level, focusing on instance pgbouncer queries and rt, table qps, query qps, etc...
* Update Dashboard: `pgsql-alert`, add Active Alert Table (you can silence alert from table panel)
* Update Dashboards:  adjust layout and content for `pgsql` dashboards.
* Dashboard Templates: Infra links are automaticlly adjusted with `nginx_upstream`
* Dashboard management script: `grafana.py`: now CI is much more easier for dashboards.
* Clickable Panel: Add data links to graphic elements.
* fix loki extraction bugs, add `unzip` to basic util set.
* fix util scripts and basic document (quick-start, download, roadmap, contribution)
* Software upgrade: grafana 8.0.5, vip-manager 1.0, prometheus 2.28.3 , consul v1.10.0, haproxy 2.2.12, pg_exporter 0.4.0
* Switch to pg_exporter v0.4.0 and new metrics set (tested in production for almost 2month, now is out of beta)
* Add time-sync option to vagrantfile
* Add new role `register` for pgsql & infra interaction
* Add new role `environ` for meta node env setup.
* Alerting rules overhaul (two implementation: Prometheus version & Grafan version)
* Remove `svc`, `role`, `ip` labels for all metrics
* ....

2021-07-09

  • now comes to the juice part, monitoring dashboard designing
  • add links between pgcat & pgsql, e.g table level dashboard
  • add a pgcat-query dashboard which aims at pg_stat_statements view
  • add alertmanager links on alert timeline panel
  • add links to graph, so user can click graphic element and jump to corresponding dashboard
  • finish pgsql-queries, and back port to pgsql query

2021-07-08

  • use [acpgh].pigsty as placeholder, passing nginx_upstream via environ, replace http host when provisioning dashboards
  • add pgsql-queries dashboard which runs on instance level, focusing on instance pgbouncer queries and rt, table qps, query qps, etc…

2021-07-07

  • Add baidu netdisk download for mainland China https://pan.baidu.com/s/1DZIa9X2jAxx69Zj-aRHoaw 8su9
  • Grafana static provision have some down-sides: root privileges / can’t update home dashboard. I wonder if we could switch to API provisioning instead.
  • Use pure python for grafana provisioning grafana.py

2021-07-06

  • Use v1.0.0-alpha1 instead. Since the change are significant, it is not appropriate to use v0.10.
  • Remove the crud haproxy index pages, using grafana table & data links instead
  • At last register by instance may be the easiest way to implement and manage
  • Add new role loki
  • Add new role promtail
  • Register datasource when create new database with pgsql-createdb.yml

2021-07-05

  • Extract a new role named register to handler all interaction between pgsql & infra.
  • Extract a new role named envrion to setup meta node environment including: ssh, metadb, env vars, etc…
  • Dashboard tags now have hierarchy: Pigsty is the top tier, Application name PGSQL PGLOG is second tier
    • Overview, Cluster, Instance,Database are filter with Pigsty and <Level> tags. which means the nav-link can cross multiple applications

2021-07-04

2021-06-30

  • Rough implementation on v0.10.0-alpha1
  • Setup environment for admin user (pgpass, pg_service, env vars,)
  • Application install script will have environ
  • Fix nofile limit on postgres|pgbouncer|patroni
  • Milestone planning.

2021-06-29

  • Remake release system
  • Have a draft on application installation standard
  • Use ‘v’ prefixed fully qualified version string
  • remove polysh from default pkg (unstable when downloading)
  • remove grafana plugins, since lot’s of them were covered in grafana 8.0

2021-06-28

  • Remake alerting rules

2021-06-25

  • Remake infra-rules and pgsql-rules

2021-06-23

  • Remake PGSQL node

2021-06-10

It’s time to have an overhaul on monitoring system, which includes:

  • Upgrade pg_exporter to 0.4.0 , re-write metric definition and add support for PostgreSQL 14
  • Use static file service discovery by default to reduce dependency for monitoring system
  • Use static label set (job,cls,ins), remove (svc,role,ip) from labels, Which makes identity immutable
  • Redesign entire monitoring system to use new label system and embrace Grafana 8.0
  • Using grafana 8.0 new features

2021-06-01

Well it’s good to write some dev logs.

51 - Why Pigsty

Historical documentation restored from the Pigsty v1.5.1 tag.

Why use Pigsty?

Our philosophy is: use the good database, use the good database so that there is no hard-to-use database in the world!

The database is the core component of an information system, a relational database is the absolute mainstay of the database, and PostgreSQL is the world’s most advanced open-source relational database.

PG provides a perfect enough database kernel, but it is not so simple to really use it well, and we help users to do that.

User’s Demand

What kind of database do traditional enterprises, especially SMEs, need for information technology? Is it a distributed cloud-native lake warehouse integrated flow batch time hyper-converged HTAP database?

No, most of the enterprise database needs, even Excel will be enough to solve! The pain point does not lie in the database kernel cattle, but whether the user can use it!

99% of enterprises, the complete life cycle of data needs, singleton PostgreSQL is sufficient!

Requirements

Software swallows the world, and open source swallows software. Cloud vendors whoring out open source, but not seeing the mantis, will eventually be dried out by multi-cloud deployments.

It’s one thing to build a personal toy demo to use a database, it’s another thing to deploy and maintain a database in a production environment: installation and deployment, operation and maintenance management, supporting facilities, platform building, service access, high availability, failover, load balancing, connection pooling, database and table splitting, monitoring, logging, auditing, backup, recovery, upgrade strategy, schema changes …… There are countless practical problems to solve, not just yum install postgresql14* && systemctl start postgresql.

PostgreSQL already provides a perfect enough database kernel, but just as Linux users are directly exposed to operating system distributions such as RedHat, SUSE, and Ubuntu, not the Linux kernel. Users need a complete solution – a database distribution, not just a database kernel.

If PostgreSQL, the database kernel, is an engine, then what users really need is the whole car, the complete, battery-included solution. What we build is such a car: stable and reliable, polished and verified in a long-time production environment; self-driving, with intelligent situational awareness.

What’s more, Pigsty is completely open-source and free! Pigsty can reduce the comprehensive cost of database ownership by 50% to 80% while providing a similar or even better experience than cloud vendors’ RDS.

Product Position

Battery-included distribution

RedHat for Linux

  • Pigsty is packaged with the PostgreSQL 14.4, the powerful geospatial plugin PostGIS 3.2, the temporal database plugin TimescaleDB 2.7, and the distributed extension plugin Citus 11.0, and hundreds of functional extensions, all installed with a single click and ready to use battery-included.
  • Pigsty integrates a complete large-scale database monitoring and control solution: Grafana, Prometheus, Loki, Ansible, CMDB, and can be used directly as a production application runtime to monitor and manage other databases and applications.
  • Pigsty integrates the most popular tools in the data analysis ecosystem: Jupyter, Echarts, Grafana, PostgREST, Postgres, and allows you to develop interactive data applications and data visualizations in a low-code manner. Produce prototypes quickly, and share, demonstrate and deliver in a standard way.

Easy-to-use Developer Toolkit

HashiCorp for Database!

  • Pigsty is designed with Infra as Data in mind, users describe what kind of database cluster they want and Pigsty automatically creates it for you! Just like Kubernetes!

  • Pigsty comes with the ultimate observability, designing monitoring systems with a BI mindset, from the topmost global insight to the most detailed every object, to get real-time data to support decision making.

  • Pigsty provides flexible and rich deployment support, local sandbox, cloud, and multi-cloud deployments. Both high-spec physical machines and 1-core 1G virtual machines can run, keeping production, pre-release, development, and test environments highly consistent.

Smart and cost-saving SRE solutions

Alternative for RDS!

  • Highly available database clusters: Pigsty integrates proven production-grade highly available database architecture solutions: master-slave offsite disaster recovery, self-healing failures, automatic high availability switchover, self-contained connection pooling, and load balancer, providing a distributed database like experience.

  • Pigsty provides a complete backup solution with one-click deployment of autopilot highly available master-slave clusters and self-healing hardware failures, greatly simplifying O&M work. Cold backup and delayed slave can effectively deal with all kinds of software failures and human failures to ensure stable system operation.

  • Pigsty can also be used as a complete SRE solution: host monitoring, application deployment, and will gradually add the deployment and monitoring of other databases: Redis/Greenplum/Kafka/Minio, or support other SaaS services, produce POC, deliver demos, etc.

VS Cloud Database RDS

Cloud Database/RDS, another “battery-included” solution, does not deliver nearly enough to satisfy the professional user:

High cost

  • The cost of RDS is 5 to 10 times higher than IDC hosting and 2 to 3 times higher even than cloud VMs.
  • The price of RDS may be advantageous relative to commercial databases, but it is still ridiculously high in front of self-build.

Life is not your choice.

  • Cloud vendors can access all types of your data, and many are not truly neutral third-party operators.
  • Cloud vendor failures are not uncommon, and the only compensation you can have is usually a poor hourly voucher.

Limited functionality

  • You don’t have true superuser access to RDS and some advanced features are not available.
  • ‘Stream replication’, and ‘high availability’ which should be standard are often sold as value-added items.

Limited experience

  • Cloud vendor RDS provides observability often with only a few sporadic monitoring metrics, lacking global integration and God’s perspective.
  • Installation, deployment, access, and use still require a lot of UI interaction and manipulation.

52 - User Interface

Historical documentation restored from the Pigsty v1.5.1 tag.

User Interface

After completing the installation, the GUI provided by Pigsty can be accessed through a browser.

http://g.pigsty -> http://10.10.10.10:80 (nginx) -> http://10.10.10.10:3000 (grafana)

Visit http://<node_ip>:3000 to browse Pigsty home (username: admin, password: pigsty)

You can visit http://demo.pigsty.cc to view the public Pigsty Demo and browse through the features provided by the Pigsty Monitoring System.

Web Services

Pigsty will provide services to the public through a series of ports, and web services will be accessed uniformly through Nginx port 80.

Component Port Default Domain Description
Grafana 3000 g.pigsty Pigsty Monitoring System GUI
Prometheus 9090 p.pigsty Monitoring Timing Database
Loki 3100 l.pigsty Log collection server (no interface)
AlertManager 9093 a.pigsty Alarm aggregation management component
Consul 8500 c.pigsty Distributed Config Management, Service Discovery
Consul DNS 8600 - Consul-provided DNS services
Nginx 80 pigsty Entry proxy for all services
Yum Repo 80 yum.pigsty Local Yum repos
Haproxy Index 123 h.pigsty Access proxy for all Haproxy management interfaces
NTP 123 n.pigsty The NTP time server used uniformly by the environment
Dnsmasq 53 - The DNS name resolution server used by the environment

Users can configure their own existing domain names for these services, or use the make dns shortcut to write the default domain names to /etc/hosts. Users can still access most services directly using the IP: Port method, for example, the entry point to the Pigsty monitoring system is the meta node IP+3000 port.

Note

Note that if Consul is used as the DCS, the Consul UI must be accessed through the Nginx domain. Consul listens on port 127.0.0.1, a deliberate design for security reasons: Consul contains sensitive metadata that should not be exposed directly to the public.

Demo

Pigsty provides a public demo at: http://demo.pigsty.cc

Because the demo instance is an empty virtual machine with 1 core and 1GB, the display is thin, so please refer to the actual effect.

53 - Deploy

Historical documentation restored from the Pigsty v1.5.1 tag.

There are 3 steps to deploy pigsty: prepare, configure, playbook

Prepare

Configure

Playbook

54 - Customization of Postgres Template

Historical documentation restored from the Pigsty v1.5.1 tag.

Parameters

The template database template1 in a cluster can be customized using the PG template configuration item.

This ensures that any database newly created in this database cluster comes with the same default configuration: schema, extensions, and default permissions.

name type hierarchy description
pg_init string G/C Custom PG initialization scripts
pg_replication_username string G PG replication user
pg_replication_password string G PG replication user’s password
pg_monitor_username string G PG monitor user
pg_monitor_password string G PG monitor user password
pg_admin_username string G PG Admin User
pg_admin_password string G PG admin user password
pg_default_roles role[] G Default created roles and users
pg_default_privileges string[] G Database default permissions configuration
pg_default_schemas string[] G The schema created by default
pg_default_extensions extension[] G Default installed extensions
pg_offline_query bool I Whether to allow offline queries
pg_reload bool A Whether to reload database configuration (HBA)
pg_hba_rules rule[] G Global HBA rules
pg_hba_rules_extra rule[] C/I Cluster/instance specific HBA rules
pgbouncer_hba_rules rule[] G/C Pgbouncer global HBA rules
pgbouncer_hba_rules_extra rule[] G/C Pgbounce specific HBA rules
pg_databases database[] G/C business database definitions
pg_users user[] G/C business user definitions

Related files

When customizing the database template, the relevant parameters are first rendered as SQL scripts and then executed on the deployed database cluster.

^---/pg/bin/pg-init
          |
          ^---(1)--- /pg/tmp/pg-init-roles.sql
          ^---(2)--- /pg/tmp/pg-init-template.sql
          ^---(3)--- <other customize logic in pg-init>

# user & database are not created during templating
^-------------(4)--- /pg/tmp/pg-user-{{ user.name }}.sql
^-------------(5)--- /pg/tmp/pg-db-{{ db.name }}.sql

pg-init

pg-init is the path to a Shell script for customizing the initialization template that will be executed as a postgres user, only on the master, when the database cluster master has been pulled up for execution, and can execute any Shell command, or execute any via psql SQL commands.

If this configuration item is not specified, Pigsty will use the default pg-init Shell script as shown below.

#!/usr/bin/env bash
set -uo pipefail


#==================================================================#
#                          Default Roles                           #
#==================================================================#
psql postgres -qAXwtf /pg/tmp/pg-init-roles.sql


#==================================================================#
#                          System Template                         #
#==================================================================#
# system default template
psql template1 -qAXwtf /pg/tmp/pg-init-template.sql

# make postgres same as templated database (optional)
psql postgres  -qAXwtf /pg/tmp/pg-init-template.sql



#==================================================================#
#                          Customize Logic                         #
#==================================================================#
# add your template logic here

This script can be appended to if the user needs to perform complex customization logic. Note pg-init is used to customize database clusters, which is usually achieved by modifying template databases. At the time this script is executed, the database cluster has been started, but the business users and business databases have not yet been created. Therefore the changes to the template database are reflected in the business database defined by default.

pg-init-roles.sql

In pg_default_roles you can customize the global unified role system. The definitions therein are rendered as /pg/tmp/pg-init-roles.sql, and a sample rendering in the pg-meta cluster is shown below.

----------------------------------------------------------------------
-- File      :   pg-init-roles.sql
-- Path      :   /pg/tmp/pg-init-roles
-- Time      :   2021-07-26 18:17
-- Note      :   managed by ansible, DO NOT CHANGE
-- Desc      :   creation sql script for default roles
----------------------------------------------------------------------


--###################################################################--
--                         dbrole_readonly                           --
--###################################################################--
-- run as dbsu (postgres by default)
-- createuser -w -p 5432 --no-login'dbrole_readonly';
-- psql -p 5432 -AXtwqf /pg/tmp/pg-user-dbrole_readonly.sql

--==================================================================--
--                           CREATE USER                            --
--==================================================================--
CREATE USER "dbrole_readonly"  NOLOGIN;

--==================================================================--
--                           ALTER USER                             --
--==================================================================--
-- options
ALTER USER "dbrole_readonly"  NOLOGIN;

-- password

-- expire

-- conn limit

-- parameters

-- comment
COMMENT ON ROLE "dbrole_readonly" IS 'role for global read-only access';


--==================================================================--
--                           GRANT ROLE                             --
--==================================================================--


--==================================================================--
--                          PGBOUNCER USER                          --
--==================================================================--
-- user will not be added to pgbouncer user list by default,
-- unless pgbouncer is explicitly set to 'true', which means production user

-- User 'dbrole_readonly' will NOT be added to /etc/pgbouncer/userlist.txt

--==================================================================--




--###################################################################--
--                         dbrole_readwrite                           --
--###################################################################--
-- run as dbsu (postgres by default)
-- createuser -w -p 5432 --no-login'dbrole_readwrite';
-- psql -p 5432 -AXtwqf /pg/tmp/pg-user-dbrole_readwrite.sql

--==================================================================--
--                           CREATE USER                            --
--==================================================================--
CREATE USER "dbrole_readwrite"  NOLOGIN;

--==================================================================--
--                           ALTER USER                             --
--==================================================================--
-- options
ALTER USER "dbrole_readwrite"  NOLOGIN;

-- password

-- expire

-- conn limit

-- parameters

-- comment
COMMENT ON ROLE "dbrole_readwrite" IS 'role for global read-write access';


--==================================================================--
--                           GRANT ROLE                             --
--==================================================================--
GRANT "dbrole_readonly" TO "dbrole_readwrite";


--==================================================================--
--                          PGBOUNCER USER                          --
--==================================================================--
-- user will not be added to pgbouncer user list by default,
-- unless pgbouncer is explicitly set to 'true', which means production user

-- User 'dbrole_readwrite' will NOT be added to /etc/pgbouncer/userlist.txt

--==================================================================--




--###################################################################--
--                         dbrole_offline                           --
--###################################################################--
-- run as dbsu (postgres by default)
-- createuser -w -p 5432 --no-login'dbrole_offline';
-- psql -p 5432 -AXtwqf /pg/tmp/pg-user-dbrole_offline.sql

--==================================================================--
--                           CREATE USER                            --
--==================================================================--
CREATE USER "dbrole_offline"  NOLOGIN;

--==================================================================--
--                           ALTER USER                             --
--==================================================================--
-- options
ALTER USER "dbrole_offline"  NOLOGIN;

-- password

-- expire

-- conn limit

-- parameters

-- comment
COMMENT ON ROLE "dbrole_offline" IS 'role for restricted read-only access (offline instance)';


--==================================================================--
--                           GRANT ROLE                             --
--==================================================================--


--==================================================================--
--                          PGBOUNCER USER                          --
--==================================================================--
-- user will not be added to pgbouncer user list by default,
-- unless pgbouncer is explicitly set to 'true', which means production user

-- User 'dbrole_offline' will NOT be added to /etc/pgbouncer/userlist.txt

--==================================================================--




--###################################################################--
--                         dbrole_admin                           --
--###################################################################--
-- run as dbsu (postgres by default)
-- createuser -w -p 5432 --no-login'dbrole_admin';
-- psql -p 5432 -AXtwqf /pg/tmp/pg-user-dbrole_admin.sql

--==================================================================--
--                           CREATE USER                            --
--==================================================================--
CREATE USER "dbrole_admin"  NOLOGIN;

--==================================================================--
--                           ALTER USER                             --
--==================================================================--
-- options
ALTER USER "dbrole_admin"  NOLOGIN;

-- password

-- expire

-- conn limit

-- parameters

-- comment
COMMENT ON ROLE "dbrole_admin" IS 'role for object creation';


--==================================================================--
--                           GRANT ROLE                             --
--==================================================================--
GRANT "pg_monitor" TO "dbrole_admin";
GRANT "dbrole_readwrite" TO "dbrole_admin";


--==================================================================--
--                          PGBOUNCER USER                          --
--==================================================================--
-- user will not be added to pgbouncer user list by default,
-- unless pgbouncer is explicitly set to 'true', which means production user

-- User 'dbrole_admin' will NOT be added to /etc/pgbouncer/userlist.txt

--==================================================================--




--###################################################################--
--                         postgres                           --
--###################################################################--
-- run as dbsu (postgres by default)
-- createuser -w -p 5432  --superuser'postgres';
-- psql -p 5432 -AXtwqf /pg/tmp/pg-user-postgres.sql

--==================================================================--
--                           CREATE USER                            --
--==================================================================--
CREATE USER "postgres"  SUPERUSER;

--==================================================================--
--                           ALTER USER                             --
--==================================================================--
-- options
ALTER USER "postgres"  SUPERUSER;

-- password

-- expire

-- conn limit

-- parameters

-- comment
COMMENT ON ROLE "postgres" IS 'system superuser';


--==================================================================--
--                           GRANT ROLE                             --
--==================================================================--


--==================================================================--
--                          PGBOUNCER USER                          --
--==================================================================--
-- user will not be added to pgbouncer user list by default,
-- unless pgbouncer is explicitly set to 'true', which means production user

-- User 'postgres' will NOT be added to /etc/pgbouncer/userlist.txt

--==================================================================--




--###################################################################--
--                         dbuser_dba                           --
--###################################################################--
-- run as dbsu (postgres by default)
-- createuser -w -p 5432  --superuser'dbuser_dba';
-- psql -p 5432 -AXtwqf /pg/tmp/pg-user-dbuser_dba.sql

--==================================================================--
--                           CREATE USER                            --
--==================================================================--
CREATE USER "dbuser_dba"  SUPERUSER;

--==================================================================--
--                           ALTER USER                             --
--==================================================================--
-- options
ALTER USER "dbuser_dba"  SUPERUSER;

-- password

-- expire

-- conn limit

-- parameters

-- comment
COMMENT ON ROLE "dbuser_dba" IS 'system admin user';


--==================================================================--
--                           GRANT ROLE                             --
--==================================================================--
GRANT "dbrole_admin" TO "dbuser_dba";


--==================================================================--
--                          PGBOUNCER USER                          --
--==================================================================--
-- user will not be added to pgbouncer user list by default,
-- unless pgbouncer is explicitly set to 'true', which means production user

-- User 'dbuser_dba' will NOT be added to /etc/pgbouncer/userlist.txt

--==================================================================--




--###################################################################--
--                         replicator                           --
--###################################################################--
-- run as dbsu (postgres by default)
-- createuser -w -p 5432  --replication'replicator';
-- psql -p 5432 -AXtwqf /pg/tmp/pg-user-replicator.sql

--==================================================================--
--                           CREATE USER                            --
--==================================================================--
CREATE USER "replicator"  REPLICATION BYPASSRLS;

--==================================================================--
--                           ALTER USER                             --
--==================================================================--
-- options
ALTER USER "replicator"  REPLICATION BYPASSRLS;

-- password

-- expire

-- conn limit

-- parameters

-- comment
COMMENT ON ROLE "replicator" IS 'system replicator';


--==================================================================--
--                           GRANT ROLE                             --
--==================================================================--
GRANT "pg_monitor" TO "replicator";
GRANT "dbrole_readonly" TO "replicator";


--==================================================================--
--                          PGBOUNCER USER                          --
--==================================================================--
-- user will not be added to pgbouncer user list by default,
-- unless pgbouncer is explicitly set to 'true', which means production user

-- User 'replicator' will NOT be added to /etc/pgbouncer/userlist.txt

--==================================================================--




--###################################################################--
--                         dbuser_monitor                           --
--###################################################################--
-- run as dbsu (postgres by default)
-- createuser -w -p 5432 'dbuser_monitor';
-- psql -p 5432 -AXtwqf /pg/tmp/pg-user-dbuser_monitor.sql

--==================================================================--
--                           CREATE USER                            --
--==================================================================--
CREATE USER "dbuser_monitor" ;

--==================================================================--
--                           ALTER USER                             --
--==================================================================--
-- options
ALTER USER "dbuser_monitor" ;

-- password

-- expire

-- conn limit

-- parameters
ALTER USER "dbuser_monitor" SET log_min_duration_statement = 1000;

-- comment
COMMENT ON ROLE "dbuser_monitor" IS 'system monitor user';


--==================================================================--
--                           GRANT ROLE                             --
--==================================================================--
GRANT "pg_monitor" TO "dbuser_monitor";
GRANT "dbrole_readonly" TO "dbuser_monitor";


--==================================================================--
--                          PGBOUNCER USER                          --
--==================================================================--
-- user will not be added to pgbouncer user list by default,
-- unless pgbouncer is explicitly set to 'true', which means production user

-- User 'dbuser_monitor' will NOT be added to /etc/pgbouncer/userlist.txt

--==================================================================--




--###################################################################--
--                         dbuser_stats                           --
--###################################################################--
-- run as dbsu (postgres by default)
-- createuser -w -p 5432 'dbuser_stats';
-- psql -p 5432 -AXtwqf /pg/tmp/pg-user-dbuser_stats.sql

--==================================================================--
--                           CREATE USER                            --
--==================================================================--
CREATE USER "dbuser_stats" ;

--==================================================================--
--                           ALTER USER                             --
--==================================================================--
-- options
ALTER USER "dbuser_stats" ;

-- password
ALTER USER "dbuser_stats" PASSWORD 'DBUser.Stats';

-- expire

-- conn limit

-- parameters

-- comment
COMMENT ON ROLE "dbuser_stats" IS 'business offline user for offline queries and ETL';


--==================================================================--
--                           GRANT ROLE                             --
--==================================================================--
GRANT "dbrole_offline" TO "dbuser_stats";


--==================================================================--
--                          PGBOUNCER USER                          --
--==================================================================--
-- user will not be added to pgbouncer user list by default,
-- unless pgbouncer is explicitly set to 'true', which means production user

-- User 'dbuser_stats' will NOT be added to /etc/pgbouncer/userlist.txt

--==================================================================--






--==================================================================--
--                       PASSWORD OVERWRITE                         --
--==================================================================--
ALTER ROLE "replicator" PASSWORD 'DBUser.Replicator';
ALTER ROLE "dbuser_monitor" PASSWORD 'DBUser.Monitor';
ALTER ROLE "dbuser_dba" PASSWORD 'DBUser.DBA';
--==================================================================--

pg-init-template.sql

PG模板参数大多会通过pg-init-template.sql的方式渲染，pg-meta集群中的渲染样例如下所示：

----------------------------------------------------------------------
-- File      :   pg-init-template.sql
-- Ctime     :   2018-10-30
-- Mtime     :   2021-02-27
-- Desc      :   init postgres cluster template
-- Path      :   /pg/tmp/pg-init-template.sql
-- Author    :   Vonng([email protected])
-- Copyright (C) 2018-2022 Ruohang Feng
----------------------------------------------------------------------


--==================================================================--
--                           Executions                             --
--==================================================================--
-- psql template1 -AXtwqf /pg/tmp/pg-init-template.sql
-- this sql scripts is responsible for post-init procedure
-- it will
--    * create system users such as replicator, monitor user, admin user
--    * create system default roles
--    * create schema, extensions in template1 & postgres
--    * create monitor views in template1 & postgres


--==================================================================--
--                          Default Privileges                      --
--==================================================================--
ALTER DEFAULT PRIVILEGES FOR ROLE postgres GRANT USAGE                         ON SCHEMAS   TO dbrole_readonly;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres GRANT SELECT                        ON TABLES    TO dbrole_readonly;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres GRANT SELECT                        ON SEQUENCES TO dbrole_readonly;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres GRANT EXECUTE                       ON FUNCTIONS TO dbrole_readonly;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres GRANT USAGE                         ON SCHEMAS   TO dbrole_offline;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres GRANT SELECT                        ON TABLES    TO dbrole_offline;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres GRANT SELECT                        ON SEQUENCES TO dbrole_offline;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres GRANT EXECUTE                       ON FUNCTIONS TO dbrole_offline;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres GRANT INSERT, UPDATE, DELETE        ON TABLES    TO dbrole_readwrite;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres GRANT USAGE, UPDATE                 ON SEQUENCES TO dbrole_readwrite;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres GRANT TRUNCATE, REFERENCES, TRIGGER ON TABLES    TO dbrole_admin;
ALTER DEFAULT PRIVILEGES FOR ROLE postgres GRANT CREATE                        ON SCHEMAS   TO dbrole_admin;

ALTER DEFAULT PRIVILEGES FOR ROLE dbuser_dba GRANT USAGE                         ON SCHEMAS   TO dbrole_readonly;
ALTER DEFAULT PRIVILEGES FOR ROLE dbuser_dba GRANT SELECT                        ON TABLES    TO dbrole_readonly;
ALTER DEFAULT PRIVILEGES FOR ROLE dbuser_dba GRANT SELECT                        ON SEQUENCES TO dbrole_readonly;
ALTER DEFAULT PRIVILEGES FOR ROLE dbuser_dba GRANT EXECUTE                       ON FUNCTIONS TO dbrole_readonly;
ALTER DEFAULT PRIVILEGES FOR ROLE dbuser_dba GRANT USAGE                         ON SCHEMAS   TO dbrole_offline;
ALTER DEFAULT PRIVILEGES FOR ROLE dbuser_dba GRANT SELECT                        ON TABLES    TO dbrole_offline;
ALTER DEFAULT PRIVILEGES FOR ROLE dbuser_dba GRANT SELECT                        ON SEQUENCES TO dbrole_offline;
ALTER DEFAULT PRIVILEGES FOR ROLE dbuser_dba GRANT EXECUTE                       ON FUNCTIONS TO dbrole_offline;
ALTER DEFAULT PRIVILEGES FOR ROLE dbuser_dba GRANT INSERT, UPDATE, DELETE        ON TABLES    TO dbrole_readwrite;
ALTER DEFAULT PRIVILEGES FOR ROLE dbuser_dba GRANT USAGE, UPDATE                 ON SEQUENCES TO dbrole_readwrite;
ALTER DEFAULT PRIVILEGES FOR ROLE dbuser_dba GRANT TRUNCATE, REFERENCES, TRIGGER ON TABLES    TO dbrole_admin;
ALTER DEFAULT PRIVILEGES FOR ROLE dbuser_dba GRANT CREATE                        ON SCHEMAS   TO dbrole_admin;

-- for additional business admin, they can SET ROLE to dbrole_admin
ALTER DEFAULT PRIVILEGES FOR ROLE "dbrole_admin" GRANT USAGE                         ON SCHEMAS   TO dbrole_readonly;
ALTER DEFAULT PRIVILEGES FOR ROLE "dbrole_admin" GRANT SELECT                        ON TABLES    TO dbrole_readonly;
ALTER DEFAULT PRIVILEGES FOR ROLE "dbrole_admin" GRANT SELECT                        ON SEQUENCES TO dbrole_readonly;
ALTER DEFAULT PRIVILEGES FOR ROLE "dbrole_admin" GRANT EXECUTE                       ON FUNCTIONS TO dbrole_readonly;
ALTER DEFAULT PRIVILEGES FOR ROLE "dbrole_admin" GRANT USAGE                         ON SCHEMAS   TO dbrole_offline;
ALTER DEFAULT PRIVILEGES FOR ROLE "dbrole_admin" GRANT SELECT                        ON TABLES    TO dbrole_offline;
ALTER DEFAULT PRIVILEGES FOR ROLE "dbrole_admin" GRANT SELECT                        ON SEQUENCES TO dbrole_offline;
ALTER DEFAULT PRIVILEGES FOR ROLE "dbrole_admin" GRANT EXECUTE                       ON FUNCTIONS TO dbrole_offline;
ALTER DEFAULT PRIVILEGES FOR ROLE "dbrole_admin" GRANT INSERT, UPDATE, DELETE        ON TABLES    TO dbrole_readwrite;
ALTER DEFAULT PRIVILEGES FOR ROLE "dbrole_admin" GRANT USAGE, UPDATE                 ON SEQUENCES TO dbrole_readwrite;
ALTER DEFAULT PRIVILEGES FOR ROLE "dbrole_admin" GRANT TRUNCATE, REFERENCES, TRIGGER ON TABLES    TO dbrole_admin;
ALTER DEFAULT PRIVILEGES FOR ROLE "dbrole_admin" GRANT CREATE                        ON SCHEMAS   TO dbrole_admin;

--==================================================================--
--                              Schemas                             --
--==================================================================--
CREATE SCHEMA IF NOT EXISTS "monitor";

-- revoke public creation
REVOKE CREATE ON SCHEMA public FROM PUBLIC;

--==================================================================--
--                             Extensions                           --
--==================================================================--
CREATE EXTENSION IF NOT EXISTS "pg_stat_statements" WITH SCHEMA "monitor";
CREATE EXTENSION IF NOT EXISTS "pgstattuple" WITH SCHEMA "monitor";
CREATE EXTENSION IF NOT EXISTS "pg_qualstats" WITH SCHEMA "monitor";
CREATE EXTENSION IF NOT EXISTS "pg_buffercache" WITH SCHEMA "monitor";
CREATE EXTENSION IF NOT EXISTS "pageinspect" WITH SCHEMA "monitor";
CREATE EXTENSION IF NOT EXISTS "pg_prewarm" WITH SCHEMA "monitor";
CREATE EXTENSION IF NOT EXISTS "pg_visibility" WITH SCHEMA "monitor";
CREATE EXTENSION IF NOT EXISTS "pg_freespacemap" WITH SCHEMA "monitor";
CREATE EXTENSION IF NOT EXISTS "pg_repack" WITH SCHEMA "monitor";
CREATE EXTENSION IF NOT EXISTS "postgres_fdw";
CREATE EXTENSION IF NOT EXISTS "file_fdw";
CREATE EXTENSION IF NOT EXISTS "btree_gist";
CREATE EXTENSION IF NOT EXISTS "btree_gin";
CREATE EXTENSION IF NOT EXISTS "pg_trgm";
CREATE EXTENSION IF NOT EXISTS "intagg";
CREATE EXTENSION IF NOT EXISTS "intarray";



--==================================================================--
--                            Monitor Views                         --
--==================================================================--

----------------------------------------------------------------------
-- cleanse
----------------------------------------------------------------------
CREATE SCHEMA IF NOT EXISTS monitor;
GRANT USAGE ON SCHEMA monitor TO "dbuser_monitor";
GRANT USAGE ON SCHEMA monitor TO "dbuser_dba";
GRANT USAGE ON SCHEMA monitor TO "replicator";

DROP VIEW IF EXISTS monitor.pg_table_bloat_human;
DROP VIEW IF EXISTS monitor.pg_index_bloat_human;
DROP VIEW IF EXISTS monitor.pg_table_bloat;
DROP VIEW IF EXISTS monitor.pg_index_bloat;
DROP VIEW IF EXISTS monitor.pg_session;
DROP VIEW IF EXISTS monitor.pg_kill;
DROP VIEW IF EXISTS monitor.pg_cancel;
DROP VIEW IF EXISTS monitor.pg_seq_scan;


----------------------------------------------------------------------
-- Table bloat estimate
----------------------------------------------------------------------
CREATE OR REPLACE VIEW monitor.pg_table_bloat AS
    SELECT CURRENT_CATALOG AS datname, nspname, relname , bs * tblpages AS size,
           CASE WHEN tblpages - est_tblpages_ff > 0 THEN (tblpages - est_tblpages_ff)/tblpages::FLOAT ELSE 0 END AS ratio
    FROM (
             SELECT ceil( reltuples / ( (bs-page_hdr)*fillfactor/(tpl_size*100) ) ) + ceil( toasttuples / 4 ) AS est_tblpages_ff,
                    tblpages, fillfactor, bs, tblid, nspname, relname, is_na
             FROM (
                      SELECT
                          ( 4 + tpl_hdr_size + tpl_data_size + (2 * ma)
                              - CASE WHEN tpl_hdr_size % ma = 0 THEN ma ELSE tpl_hdr_size % ma END
                              - CASE WHEN ceil(tpl_data_size)::INT % ma = 0 THEN ma ELSE ceil(tpl_data_size)::INT % ma END
                              ) AS tpl_size, (heappages + toastpages) AS tblpages, heappages,
                          toastpages, reltuples, toasttuples, bs, page_hdr, tblid, nspname, relname, fillfactor, is_na
                      FROM (
                               SELECT
                                   tbl.oid AS tblid, ns.nspname , tbl.relname, tbl.reltuples,
                                   tbl.relpages AS heappages, coalesce(toast.relpages, 0) AS toastpages,
                                   coalesce(toast.reltuples, 0) AS toasttuples,
                                   coalesce(substring(array_to_string(tbl.reloptions, ' ') FROM 'fillfactor=([0-9]+)')::smallint, 100) AS fillfactor,
                                   current_setting('block_size')::numeric AS bs,
                                   CASE WHEN version()~'mingw32' OR version()~'64-bit|x86_64|ppc64|ia64|amd64' THEN 8 ELSE 4 END AS ma,
                                   24 AS page_hdr,
                                   23 + CASE WHEN MAX(coalesce(s.null_frac,0)) > 0 THEN ( 7 + count(s.attname) ) / 8 ELSE 0::int END
                                       + CASE WHEN bool_or(att.attname = 'oid' and att.attnum < 0) THEN 4 ELSE 0 END AS tpl_hdr_size,
                                   sum( (1-coalesce(s.null_frac, 0)) * coalesce(s.avg_width, 0) ) AS tpl_data_size,
                                   bool_or(att.atttypid = 'pg_catalog.name'::regtype)
                                       OR sum(CASE WHEN att.attnum > 0 THEN 1 ELSE 0 END) <> count(s.attname) AS is_na
                               FROM pg_attribute AS att
                                        JOIN pg_class AS tbl ON att.attrelid = tbl.oid
                                        JOIN pg_namespace AS ns ON ns.oid = tbl.relnamespace
                                        LEFT JOIN pg_stats AS s ON s.schemaname=ns.nspname AND s.tablename = tbl.relname AND s.inherited=false AND s.attname=att.attname
                                        LEFT JOIN pg_class AS toast ON tbl.reltoastrelid = toast.oid
                               WHERE NOT att.attisdropped AND tbl.relkind = 'r' AND nspname NOT IN ('pg_catalog','information_schema')
                               GROUP BY 1,2,3,4,5,6,7,8,9,10
                           ) AS s
                  ) AS s2
         ) AS s3
    WHERE NOT is_na;
COMMENT ON VIEW monitor.pg_table_bloat IS 'postgres table bloat estimate';

----------------------------------------------------------------------
-- Index bloat estimate
----------------------------------------------------------------------
CREATE OR REPLACE VIEW monitor.pg_index_bloat AS
    SELECT CURRENT_CATALOG AS datname, nspname, idxname AS relname, relpages::BIGINT * bs AS size,
           COALESCE((relpages - ( reltuples * (6 + ma - (CASE WHEN index_tuple_hdr % ma = 0 THEN ma ELSE index_tuple_hdr % ma END)
                                                   + nulldatawidth + ma - (CASE WHEN nulldatawidth % ma = 0 THEN ma ELSE nulldatawidth % ma END))
                                      / (bs - pagehdr)::FLOAT  + 1 )), 0) / relpages::FLOAT AS ratio
    FROM (
             SELECT nspname,
                    idxname,
                    reltuples,
                    relpages,
                    current_setting('block_size')::INTEGER                                                               AS bs,
                    (CASE WHEN version() ~ 'mingw32' OR version() ~ '64-bit|x86_64|ppc64|ia64|amd64' THEN 8 ELSE 4 END)  AS ma,
                    24                                                                                                   AS pagehdr,
                    (CASE WHEN max(COALESCE(pg_stats.null_frac, 0)) = 0 THEN 2 ELSE 6 END)                               AS index_tuple_hdr,
                    sum((1.0 - COALESCE(pg_stats.null_frac, 0.0)) *
                        COALESCE(pg_stats.avg_width, 1024))::INTEGER                                                     AS nulldatawidth
             FROM pg_attribute
                      JOIN (
                 SELECT pg_namespace.nspname,
                        ic.relname                                                   AS idxname,
                        ic.reltuples,
                        ic.relpages,
                        pg_index.indrelid,
                        pg_index.indexrelid,
                        tc.relname                                                   AS tablename,
                        regexp_split_to_table(pg_index.indkey::TEXT, ' ') :: INTEGER AS attnum,
                        pg_index.indexrelid                                          AS index_oid
                 FROM pg_index
                          JOIN pg_class ic ON pg_index.indexrelid = ic.oid
                          JOIN pg_class tc ON pg_index.indrelid = tc.oid
                          JOIN pg_namespace ON pg_namespace.oid = ic.relnamespace
                          JOIN pg_am ON ic.relam = pg_am.oid
                 WHERE pg_am.amname = 'btree' AND ic.relpages > 0 AND nspname NOT IN ('pg_catalog', 'information_schema')
             ) ind_atts ON pg_attribute.attrelid = ind_atts.indexrelid AND pg_attribute.attnum = ind_atts.attnum
                      JOIN pg_stats ON pg_stats.schemaname = ind_atts.nspname
                 AND ((pg_stats.tablename = ind_atts.tablename AND pg_stats.attname = pg_get_indexdef(pg_attribute.attrelid, pg_attribute.attnum, TRUE))
                     OR (pg_stats.tablename = ind_atts.idxname AND pg_stats.attname = pg_attribute.attname))
             WHERE pg_attribute.attnum > 0
             GROUP BY 1, 2, 3, 4, 5, 6
         ) est
    LIMIT 512;
COMMENT ON VIEW monitor.pg_index_bloat IS 'postgres index bloat estimate (btree-only)';

----------------------------------------------------------------------
-- table bloat pretty
----------------------------------------------------------------------
CREATE OR REPLACE VIEW monitor.pg_table_bloat_human AS
SELECT nspname || '.' || relname AS name,
       pg_size_pretty(size)      AS size,
       pg_size_pretty((size * ratio)::BIGINT) AS wasted,
       round(100 * ratio::NUMERIC, 2)  as ratio
FROM monitor.pg_table_bloat ORDER BY wasted DESC NULLS LAST;
COMMENT ON VIEW monitor.pg_table_bloat_human IS 'postgres table bloat pretty';

----------------------------------------------------------------------
-- index bloat pretty
----------------------------------------------------------------------
CREATE OR REPLACE VIEW monitor.pg_index_bloat_human AS
SELECT nspname || '.' || relname              AS name,
       pg_size_pretty(size)                   AS size,
       pg_size_pretty((size * ratio)::BIGINT) AS wasted,
       round(100 * ratio::NUMERIC, 2)         as ratio
FROM monitor.pg_index_bloat;
COMMENT ON VIEW monitor.pg_index_bloat_human IS 'postgres index bloat pretty';


----------------------------------------------------------------------
-- pg session
----------------------------------------------------------------------
CREATE OR REPLACE VIEW monitor.pg_session AS
SELECT coalesce(datname, 'all') AS datname,
       numbackends,
       active,
       idle,
       ixact,
       max_duration,
       max_tx_duration,
       max_conn_duration
FROM (
         SELECT datname,
                count(*)                                         AS numbackends,
                count(*) FILTER ( WHERE state = 'active' )       AS active,
                count(*) FILTER ( WHERE state = 'idle' )         AS idle,
                count(*) FILTER ( WHERE state = 'idle in transaction'
                    OR state = 'idle in transaction (aborted)' ) AS ixact,
                max(extract(epoch from now() - state_change))
                FILTER ( WHERE state = 'active' )                AS max_duration,
                max(extract(epoch from now() - xact_start))      AS max_tx_duration,
                max(extract(epoch from now() - backend_start))   AS max_conn_duration
         FROM pg_stat_activity
         WHERE backend_type = 'client backend'
           AND pid <> pg_backend_pid()
         GROUP BY ROLLUP (1)
         ORDER BY 1 NULLS FIRST
     ) t;
COMMENT ON VIEW monitor.pg_session IS 'postgres session stats';


----------------------------------------------------------------------
-- pg kill
----------------------------------------------------------------------
CREATE OR REPLACE VIEW monitor.pg_kill AS
SELECT pid,
       pg_terminate_backend(pid)                 AS killed,
       datname                                   AS dat,
       usename                                   AS usr,
       application_name                          AS app,
       client_addr                               AS addr,
       state,
       extract(epoch from now() - state_change)  AS query_time,
       extract(epoch from now() - xact_start)    AS xact_time,
       extract(epoch from now() - backend_start) AS conn_time,
       substring(query, 1, 40)                   AS query
FROM pg_stat_activity
WHERE backend_type = 'client backend'
  AND pid <> pg_backend_pid();
COMMENT ON VIEW monitor.pg_kill IS 'kill all backend session';


----------------------------------------------------------------------
-- quick cancel view
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_cancel;
CREATE OR REPLACE VIEW monitor.pg_cancel AS
SELECT pid,
       pg_cancel_backend(pid)                    AS cancel,
       datname                                   AS dat,
       usename                                   AS usr,
       application_name                          AS app,
       client_addr                               AS addr,
       state,
       extract(epoch from now() - state_change)  AS query_time,
       extract(epoch from now() - xact_start)    AS xact_time,
       extract(epoch from now() - backend_start) AS conn_time,
       substring(query, 1, 40)
FROM pg_stat_activity
WHERE state = 'active'
  AND backend_type = 'client backend'
  and pid <> pg_backend_pid();
COMMENT ON VIEW monitor.pg_cancel IS 'cancel backend queries';


----------------------------------------------------------------------
-- seq scan
----------------------------------------------------------------------
DROP VIEW IF EXISTS monitor.pg_seq_scan;
CREATE OR REPLACE VIEW monitor.pg_seq_scan AS
SELECT schemaname                             AS nspname,
       relname,
       seq_scan,
       seq_tup_read,
       seq_tup_read / seq_scan                AS seq_tup_avg,
       idx_scan,
       n_live_tup + n_dead_tup                AS tuples,
       n_live_tup / (n_live_tup + n_dead_tup) AS dead_ratio
FROM pg_stat_user_tables
WHERE seq_scan > 0
  and (n_live_tup + n_dead_tup) > 0
ORDER BY seq_tup_read DESC
LIMIT 50;
COMMENT ON VIEW monitor.pg_seq_scan IS 'table that have seq scan';


----------------------------------------------------------------------
-- pg_shmem auxiliary function
-- PG 13 ONLY!
----------------------------------------------------------------------
CREATE OR REPLACE FUNCTION monitor.pg_shmem() RETURNS SETOF
    pg_shmem_allocations AS $$ SELECT * FROM pg_shmem_allocations;$$ LANGUAGE SQL SECURITY DEFINER;
COMMENT ON FUNCTION monitor.pg_shmem() IS 'security wrapper for pg_shmem';


--==================================================================--
--                          Customize Logic                         --
--==================================================================--
-- This script will be execute on primary instance among a newly created
-- postgres cluster. it will be executed as dbsu on template1 database
-- put your own customize logic here
-- make sure they are idempotent

Custom Patroni templates

Pigsty uses Patroni to manage and initialize Postgres database clusters.

Pigsty uses Patroni for the main work of provisioning, even if the user selects no Patroni mode, pulling up the database cluster will be taken care of by Patroni and removing the Patroni component after the creation is complete.

Users can do most of the PostgreSQL cluster customization through Patroni configuration files. For details of Patroni configuration file format, please refer to Patroni official documentation.

Predefined templates

Pigsty provides four predefined initialization templates, the initialization templates are the definition files used to initialize the database cluster and are located by default in roles/postgres/templates/. Included are.

  • oltp.yml OLTP template, default configuration, optimized for latency and performance for production models.
  • olap.yml OLAP template, improve parallelism, optimize for throughput, long queries.
  • crit.yml) Core business template, based on OLTP template optimized for RPO, security, data integrity, enable synchronous replication with data checksum.
  • tiny.yml Micro database template optimized for low-resource scenarios, such as demo database clusters running in virtual machines.

Specify the path to the template to be used via the pg_conf parameter, or simply fill in the template file name if using a pre-built template.

If a custom Patroni configuration template is used, the companion node optimization template should usually be used for the machine node as well.

Sample Patroni Template

When customizing your own Patroni template, you can use several existing base templates as a baseline to build upon.

and place them in the templates/ directory, just name them in mode.yml format.

Please keep the template variables in Patroni, otherwise the related parameters may not work properly.

A typical Patroni configuration file (OLTP)

#!/usr/bin/env patroni
#==============================================================#
# File      :   patroni.yml
# Ctime     :   2020-04-08
# Mtime     :   2020-12-22
# Desc      :   patroni cluster definition for {{ pg_cluster }} (oltp)
# Path      :   /pg/bin/patroni.yml
# Real Path :   /pg/conf/{{ pg_instance }}.yml
# Link      :   /pg/bin/patroni.yml -> /pg/conf/{{ pg_instance}}.yml
# Note      :   Transactional Database Cluster Template
# Doc       :   https://patroni.readthedocs.io/en/latest/SETTINGS.html
# Copyright (C) 2018-2022 Ruohang Feng
#==============================================================#

# OLTP database are optimized for performance, rt latency
# typical spec: 64 Core | 400 GB RAM | PCI-E SSD xTB

---
#------------------------------------------------------------------------------
# identity
#------------------------------------------------------------------------------
namespace: {{ pg_namespace }}/          # namespace
scope: {{ pg_cluster }}                 # cluster name
name: {{ pg_instance }}                 # instance name

#------------------------------------------------------------------------------
# log
#------------------------------------------------------------------------------
log:
  level: INFO                           #  NOTEST|DEBUG|INFO|WARNING|ERROR|CRITICAL
  dir: /pg/log/                         #  default log file: /pg/log/patroni.log
  file_size: 33554432                   #  32MB log triggers log rotation
  file_num: 20                          #  keep at most 30x32MB = 1GB log
  dateformat: '%Y-%m-%d %H:%M:%S %z'    #  IMPORTANT: discard milli timestamp
  format: '%(asctime)s %(levelname)s: %(message)s'

#------------------------------------------------------------------------------
# dcs
#------------------------------------------------------------------------------
consul:
  host: 127.0.0.1:8500
  consistency: default         # default|consistent|stale
  register_service: true
  service_check_interval: 15s
  service_tags:
    - {{ pg_cluster }}

#------------------------------------------------------------------------------
# api
#------------------------------------------------------------------------------
# how to expose patroni service
# listen on all ipv4, connect via public ip, use same credential as dbuser_monitor
restapi:
  listen: 0.0.0.0:{{ patroni_port }}
  connect_address: {{ inventory_hostname }}:{{ patroni_port }}
  authentication:
    verify_client: none                 # none|optional|required
    username: {{ pg_monitor_username }}
    password: '{{ pg_monitor_password }}'

#------------------------------------------------------------------------------
# ctl
#------------------------------------------------------------------------------
ctl:
  optional:
    insecure: true
    # cacert: '/path/to/ca/cert'
    # certfile: '/path/to/cert/file'
    # keyfile: '/path/to/key/file'

#------------------------------------------------------------------------------
# tags
#------------------------------------------------------------------------------
tags:
  nofailover: false
  clonefrom: true
  noloadbalance: false
  nosync: false
{% if pg_upstream is defined %}
  replicatefrom: {{ pg_upstream }}    # clone from another replica rather than primary
{% endif %}

#------------------------------------------------------------------------------
# watchdog
#------------------------------------------------------------------------------
# available mode: off|automatic|required
watchdog:
  mode: {{ patroni_watchdog_mode }}
  device: /dev/watchdog
  # safety_margin: 10s

#------------------------------------------------------------------------------
# bootstrap
#------------------------------------------------------------------------------
bootstrap:

  #----------------------------------------------------------------------------
  # bootstrap method
  #----------------------------------------------------------------------------
  method: initdb
  # add custom bootstrap method here

  # default bootstrap method: initdb
  initdb:
{% if pg_encoding != '' %}
    - encoding: {{ pg_encoding }}
{% endif %}
{% if pg_locale != '' %}
    - locale: {{ pg_locale }}
{% endif %}
{% if pg_lc_collate != '' %}
    - lc-collate: {{ pg_lc_collate }}
{% endif %}
{% if pg_lc_ctype != '' %}
    - lc-ctype: {{ pg_lc_ctype }}
{% endif %}

  #----------------------------------------------------------------------------
  # bootstrap users
  #---------------------------------------------------------------------------
  # additional users which need to be created after initializing new cluster
  # replication user and monitor user are required
  users:
    {{ pg_replication_username }}:
      password: '{{ pg_replication_password }}'
    {{ pg_monitor_username }}:
      password: '{{ pg_monitor_password }}'
    {{ pg_admin_username }}:
      password: '{{ pg_admin_password }}'

  # bootstrap hba, allow local and intranet password access & replication
  # will be overwritten later
  pg_hba:
    - local   all             postgres                                ident
    - local   all             all                                     md5
    - host    all             all            0.0.0.0/0                md5
    - local   replication     postgres                                ident
    - local   replication     all                                     md5
    - host    replication     all            0.0.0.0/0                md5


  #----------------------------------------------------------------------------
  # template
  #---------------------------------------------------------------------------
  # post_init: /pg/bin/pg-init

  #----------------------------------------------------------------------------
  # bootstrap config
  #---------------------------------------------------------------------------
  # this section will be written to /{{ pg_namespace }}/{{ pg_cluster }}/config
  # if will NOT take any effect after cluster bootstrap
  dcs:

{% if pg_role == 'primary' and pg_upstream is defined %}
    #----------------------------------------------------------------------------
    # standby cluster definition
    #---------------------------------------------------------------------------
    standby_cluster:
      host: {{ pg_upstream }}
      port: {{ pg_port }}
      # primary_slot_name: patroni     # must be create manually on upstream server, if specified
      create_replica_methods:
        - basebackup
{% endif %}

    #----------------------------------------------------------------------------
    # important parameters
    #---------------------------------------------------------------------------
    # constraint: ttl >: loop_wait + retry_timeout * 2

    # the number of seconds the loop will sleep. Default value: 10
    # this is patroni check loop interval
    loop_wait: 10

    # the TTL to acquire the leader lock (in seconds). Think of it as the length of time before initiation of the automatic failover process. Default value: 30
    # config this according to your network condition to avoid false-positive failover
    ttl: 30

    # timeout for DCS and PostgreSQL operation retries (in seconds). DCS or network issues shorter than this will not cause Patroni to demote the leader. Default value: 10
    retry_timeout: 10

    # the amount of time a master is allowed to recover from failures before failover is triggered (in seconds)
    # Max RTO: 2 loop wait + master_start_timeout
    master_start_timeout: 10

    # import: candidate will not be promoted if replication lag is higher than this
    # maximum RPO: 1MB
    maximum_lag_on_failover: 1048576

    # The number of seconds Patroni is allowed to wait when stopping Postgres and effective only when synchronous_mode is enabled
    master_stop_timeout: 30

    # turns on synchronous replication mode. In this mode a replica will be chosen as synchronous and only the latest leader and synchronous replica are able to participate in leader election
    # set to true for RPO mode
    synchronous_mode: false

    # prevents disabling synchronous replication if no synchronous replicas are available, blocking all client writes to the master
    synchronous_mode_strict: false


    #----------------------------------------------------------------------------
    # postgres parameters
    #---------------------------------------------------------------------------
    postgresql:
      use_slots: true
      use_pg_rewind: true
      remove_data_directory_on_rewind_failure: true


      parameters:
        #----------------------------------------------------------------------
        # IMPORTANT PARAMETERS
        #----------------------------------------------------------------------
        max_connections: 800                    # 100 -> 800
        superuser_reserved_connections: 10      # reserve 10 connection for su
        max_locks_per_transaction: 128          # 64 -> 128
        max_prepared_transactions: 0            # 0 disable 2PC
        track_commit_timestamp: on              # enabled xact timestamp
        max_worker_processes: 64                # default 8 -> 64, set to cpu core 64
        wal_level: logical                      # logical
        wal_log_hints: on                       # wal log hints to support rewind
        max_wal_senders: 24                     # 10 -> 24
        max_replication_slots: 16               # 10 -> 16
        wal_keep_size: 100GB                    # keep at least 100GB WAL
        password_encryption: md5                # use traditional md5 auth

        #----------------------------------------------------------------------
        # RESOURCE USAGE (except WAL)
        #----------------------------------------------------------------------
        # memory: shared_buffers and maintenance_work_mem will be dynamically set
        shared_buffers: {{ pg_shared_buffers }}
        maintenance_work_mem: {{ pg_maintenance_work_mem }}
        work_mem: 32MB                          # 4MB -> 32MB
        huge_pages: try                         # try huge pages
        temp_file_limit: 100GB                  # 0 -> 100GB
        vacuum_cost_delay: 2ms                  # wait 2ms per 10000 cost
        vacuum_cost_limit: 10000                # 10000 cost each round
        bgwriter_delay: 10ms                    # check dirty page every 10ms
        bgwriter_lru_maxpages: 800              # 100 -> 800
        bgwriter_lru_multiplier: 5.0            # 2.0 -> 5.0  more cushion buffer
        max_parallel_workers: 32                # default 8 -> 32, limit by max_worker_processes
        max_parallel_maintenance_workers: 8     # default 2 -> 8, limit by parallel worker
        max_parallel_workers_per_gather: 0      # default 2 -> 0, disable parallel query in OLTP mode

        #----------------------------------------------------------------------
        # WAL
        #----------------------------------------------------------------------
        wal_buffers: 16MB                       # max to 16MB
        wal_writer_delay: 20ms                  # wait period
        wal_writer_flush_after: 1MB             # max allowed data loss
        min_wal_size: 100GB                     # at least 100GB WAL
        max_wal_size: 400GB                     # at most 400GB WAL
        commit_delay: 20                        # 200ms -> 20ms, increase speed
        commit_siblings: 10                     # 5 -> 10
        checkpoint_timeout: 60min               # checkpoint 5min -> 1h
        checkpoint_completion_target: 0.95      # 0.5 -> 0.95
        archive_mode: on
        archive_command: 'wal_dir=/pg/arcwal; [[ $(date +%H%M) == 1200 ]] && rm -rf ${wal_dir}/$(date -d"yesterday" +%Y%m%d); /bin/mkdir -p ${wal_dir}/$(date +%Y%m%d) && /usr/bin/lz4 -q -z %p > ${wal_dir}/$(date +%Y%m%d)/%f.lz4'

        #----------------------------------------------------------------------
        # REPLICATION
        #----------------------------------------------------------------------
        # synchronous_standby_names: ''
        vacuum_defer_cleanup_age: 50000         # 0->50000 last 50000 xact changes will not be vacuumed
        promote_trigger_file: promote.signal    # default promote trigger file path
        max_standby_archive_delay: 10min        # max delay before canceling queries when reading WAL from archive;
        max_standby_streaming_delay: 3min       # max delay before canceling queries when reading streaming WAL;
        wal_receiver_status_interval: 1s        # send replies at least this often
        hot_standby_feedback: on                # send info from standby to prevent query conflicts
        wal_receiver_timeout: 60s               # time that receiver waits for
        max_logical_replication_workers: 8      # 4 -> 8, 6 sync worker + 1~2 apply worker
        max_sync_workers_per_subscription: 6    # 2 -> 6, 6 sync worker

        #----------------------------------------------------------------------
        # QUERY TUNING
        #----------------------------------------------------------------------
        # planner
        # enable_partitionwise_join: on
        random_page_cost: 1.1                   # 4 for HDD, 1.1 for SSD
        effective_cache_size: 320GB             # max mem - shared buffer
        default_statistics_target: 1000         # stat bucket 100 -> 1000

        #----------------------------------------------------------------------
        # REPORTING AND LOGGING
        #----------------------------------------------------------------------
        log_destination: csvlog                 # use standard csv log
        logging_collector: on                   # enable csvlog
        log_directory: log                      # default log dir: /pg/data/log
        # log_filename: 'postgresql-%a.log'     # weekly auto-recycle
        log_filename: 'postgresql-%Y-%m-%d.log' # YYYY-MM-DD full log retention
        log_checkpoints: on                     # log checkpoint info
        log_lock_waits: on                      # log lock wait info
        log_replication_commands: on            # log replication info
        log_statement: ddl                      # log ddl change
        log_min_duration_statement: 100         # log slow query (>100ms)

        #----------------------------------------------------------------------
        # STATISTICS
        #----------------------------------------------------------------------
        track_io_timing: on                     # collect io statistics
        track_functions: all                    # track all functions (none|pl|all)
        track_activity_query_size: 8192         # max query length in pg_stat_activity

        #----------------------------------------------------------------------
        # AUTOVACUUM
        #----------------------------------------------------------------------
        log_autovacuum_min_duration: 1s         # log autovacuum activity take more than 1s
        autovacuum_max_workers: 3               # default autovacuum worker 3
        autovacuum_naptime: 1min                # default autovacuum naptime 1min
        autovacuum_vacuum_scale_factor: 0.08    # fraction of table size before vacuum   20% -> 8%
        autovacuum_analyze_scale_factor: 0.04   # fraction of table size before analyze  10% -> 4%
        autovacuum_vacuum_cost_delay: -1        # default vacuum cost delay: same as vacuum_cost_delay
        autovacuum_vacuum_cost_limit: -1        # default vacuum cost limit: same as vacuum_cost_limit
        autovacuum_freeze_max_age: 1000000000   # age > 1 billion triggers force vacuum

        #----------------------------------------------------------------------
        # CLIENT
        #----------------------------------------------------------------------
        deadlock_timeout: 50ms                  # 50ms for deadlock
        idle_in_transaction_session_timeout: 10min  # 10min timeout for idle in transaction

        #----------------------------------------------------------------------
        # CUSTOMIZED OPTIONS
        #----------------------------------------------------------------------
        # extensions
        shared_preload_libraries: '{{ pg_libs | default("pg_stat_statements, auto_explain") }}'

        # auto_explain
        auto_explain.log_min_duration: 1s       # auto explain query slower than 1s
        auto_explain.log_analyze: true          # explain analyze
        auto_explain.log_verbose: true          # explain verbose
        auto_explain.log_timing: true           # explain timing
        auto_explain.log_nested_statements: true

        # pg_stat_statements
        pg_stat_statements.max: 10000           # 5000 -> 10000 queries
        pg_stat_statements.track: all           # track all statements (all|top|none)
        pg_stat_statements.track_utility: off   # do not track query other than CRUD
        pg_stat_statements.track_planning: off  # do not track planning metrics


#------------------------------------------------------------------------------
# postgres
#------------------------------------------------------------------------------
postgresql:

  #----------------------------------------------------------------------------
  # how to connect to postgres
  #----------------------------------------------------------------------------
  bin_dir: {{ pg_bin_dir }}
  data_dir: {{ pg_data }}
  config_dir: {{ pg_data }}
  pgpass: {{ pg_dbsu_home }}/.pgpass
  listen: {{ pg_listen }}:{{ pg_port }}
  connect_address: {{ inventory_hostname }}:{{ pg_port }}
  use_unix_socket: true # default: /var/run/postgresql, /tmp

  #----------------------------------------------------------------------------
  # who to connect to postgres
  #----------------------------------------------------------------------------
  authentication:
    superuser:
      username: {{ pg_dbsu }}
    replication:
      username: {{ pg_replication_username }}
      password: '{{ pg_replication_password }}'
    rewind:
      username: {{ pg_replication_username }}
      password: '{{ pg_replication_password }}'

  #----------------------------------------------------------------------------
  # how to react to database operations
  #----------------------------------------------------------------------------
  # event callback script log: /pg/log/callback.log
  callbacks:
    on_start: /pg/bin/pg-failover-callback
    on_stop: /pg/bin/pg-failover-callback
    on_reload: /pg/bin/pg-failover-callback
    on_restart: /pg/bin/pg-failover-callback
    on_role_change: /pg/bin/pg-failover-callback

  # rewind policy: data checksum should be enabled before using rewind
  use_pg_rewind: true
  remove_data_directory_on_rewind_failure: true
  remove_data_directory_on_diverged_timelines: false

  #----------------------------------------------------------------------------
  # how to create replica
  #----------------------------------------------------------------------------
  # create replica method: default pg_basebackup
  create_replica_methods:
    - basebackup
  basebackup:
    - max-rate: '1000M'
    - checkpoint: fast
    - status-interva: 1s
    - verbose
    - progress

  #----------------------------------------------------------------------------
  # ad hoc parameters (overwrite with default)
  #----------------------------------------------------------------------------
  # parameters:

  #----------------------------------------------------------------------------
  # host based authentication, overwrite default pg_hba.conf
  #----------------------------------------------------------------------------
  # pg_hba:
  #   - local   all             postgres                                ident
  #   - local   all             all                                     md5
  #   - host    all             all            0.0.0.0/0                md5
  #   - local   replication     postgres                                ident
  #   - local   replication     all                                     md5
  #   - host    replication     all            0.0.0.0/0                md5

...

55 - Redis Deploy & Monitor

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty is also a universal application runtime that could be used for deploying & monitoring other databases and applications, such as Redis.

It takes two steps to deploy a Redis cluster:

  1. Declare it
  2. Execute playbook

Define Redis

E-R Model

The Redis entity conceptual model is almost identical to [PostgreSQL]c-pgsql.md#ER-Model), and also includes the concepts of Cluster and Instance. Note that the concept of Cluster here does not refer to the clusters in Redis’ native clustering scheme.

The core difference is that Redis is typically deployed in a single multi-instance deployment, with many Redis instances typically deployed on a single physical/virtual machine node to take advantage of multi-core CPUs. therefore, how Redis instances are defined is slightly different from PGSQL.

In Pigsty-managed Redis, the nodes are fully subordinate to the cluster, i.e. it is not currently allowed to deploy two Redis instances from different clusters on a single node, but this does not prevent you from deploying multiple independent Redis instances on a single node.

Redis Identity

The identity parameters are the information that must be provided when defining a Redis cluster and include.

name attributes description example
redis_cluster must, cluster level cluster-name redis-test
redis_node must, node level node number 1,2
redis_instances MUST, node level instance definition { 6001 : {} ,6002 : {}}
  • redis_cluster identifies the name of the Redis cluster, configured at the cluster level, and serves as the top-level namespace for cluster resources.
  • redis_node identifies the serial number of the node in the cluster
  • redis_instances is a JSON object with the Key as the instance port number and the Value as a JSON object containing the instance-specific configuration

Redis Cluster Definition

Given below are three condensed definitions of Redis clusters, including.

  • A 1-node, 3-instance Redis Sentinel cluster redis-sentinel
  • A 2-node, 12-instance Redis Cluster redis-cluster
  • A 1-node, one-master-two-slave Redis Standalone cluster redis-standalone

You need to assign a unique port number to the Redis instance on the node.

#----------------------------------#
# sentinel example                 #
#----------------------------------#
redis-sentinel:
  hosts:
    10.10.10.10:
      redis_node: 1
      redis_instances:  { 6001 : {} ,6002 : {} , 6003 : {} }
  vars:
    redis_cluster: redis-sentinel
    redis_mode: sentinel
    redis_max_memory: 128MB

#----------------------------------#
# cluster example                  #
#----------------------------------#
redis-cluster:
  hosts:
    10.10.10.11:
      redis_node: 1
      redis_instances: { 6501 : {} ,6502 : {} ,6503 : {} ,6504 : {} ,6505 : {} ,6506 : {} }
    10.10.10.12:
      redis_node: 2
      redis_instances: { 6501 : {} ,6502 : {} ,6503 : {} ,6504 : {} ,6505 : {} ,6506 : {} }
  vars:
    redis_cluster: redis-cluster        # name of this redis 'cluster'
    redis_mode: cluster                 # standalone,cluster,sentinel
    redis_max_memory: 64MB              # max memory used by each redis instance
    redis_mem_policy: allkeys-lru       # memory eviction policy

#----------------------------------#
# standalone example               #
#----------------------------------#
redis-standalone:
  hosts:
    10.10.10.13:
      redis_node: 1
      redis_instances:
        6501: {}
        6502: { replica_of: '10.10.10.13 6501' }
        6503: { replica_of: '10.10.10.13 6501' }
  vars:
    redis_cluster: redis-standalone     # name of this redis 'cluster'
    redis_mode: standalone              # standalone,cluster,sentinel
    redis_max_memory: 64MB              # max memory used by each redis instance

Create Cluster

Deployment script

Create a Redis instance/cluster using the script redis.yml

./redis.yml -l redis-sentinel
./redis.yml -l redis-cluster
./redis.yml -l redis-standalone

Notes

Although this is not the recommended behavior, you can deploy PostgreSQL with Redis in a mixed deployment to make the most of machine resources.

The redis.yml script will deploy the Redis Monitor Exporter on the machine at the same time, including redis_exporter and node_exporter (optional)

During this process, the machine’s node_exporter will be redeployed if it exists.

By default, Prometheus will use the “multi-target crawl” mode, using the Redis Exporter on port 9121 on the node to crawl all Redis instances on that node.

Redis Monitoring

Pigsty currently provides 3 Redis monitoring panels as part of a standalone monitoring application REDIS, which are.

  • Redis Overview: provides a global overview of Redis across the entire environment
  • Redis Cluster: focuses on monitoring information for a single Redis business cluster
  • Redis Instance: provides detailed monitoring information about a single Redis instance

You can use the included redis-benchmark test

CAVEAT

Pigsty v1.5.1 supports whole-cluster Redis deployment and monitoring, and the tagged redis.yml/redis-remove.yml playbooks also target one instance with -e redis_port=<port>.

Offline, scale-up, scale-down, and single-instance management features will be provided in subsequent versions gradually.

56 - PGWeb

Historical documentation restored from the Pigsty v1.5.1 tag.

Pigsty v1.5.1 does not ship the former infra-pgweb.yml playbook or the pgweb_enabled / pgweb_username inventory parameters. PGWeb is provided as a Docker application at app/pgweb.

Run PGWeb

Docker is enabled on the meta node in the bundled v1.5.1 inventory. Start the tagged PGWeb application with:

cd ~/pigsty/app/pgweb
docker-compose up -d

The tagged Compose file maps host port 8886 to container port 8081. Supply a database connection string in the PGWeb interface, for example postgres://dbuser_dba:[email protected]:5432/meta after replacing the credentials as appropriate.

To expose it through Nginx, add an nginx_upstreams entry for 127.0.0.1:8886 and rerun ./infra.yml -t nginx_config,nginx_restart. See Docker Applications for the frozen v1.5.1 application context.

57 - Using TimescaleDB to store Prometheus data

Historical documentation restored from the Pigsty v1.5.1 tag.

You can use postgres as the remote storage database used by the Prometheus backend.

While this is not the recommended behavior, it is a good opportunity to understand how the Pigsty deployment system is used.

Preparing the Postgres database 

vi pigsty.yml # Uncomment DB/User definition: dbuser_prometheus prometheus

pg_databases:                           # define business users/roles on this cluster, array of user definition
  - { name: prometheus, owner: dbuser_prometheus , revokeconn: true, comment: prometheus primary database }
pg_users:                           # define business users/roles on this cluster, array of user definition
  - {name: dbuser_prometheus , password: DBUser.Prometheus ,pgbouncer: true , createrole: true,  roles: [dbrole_admin], comment: admin user for prometheus database }

Create a Prometheus business database with business users.

bin/createuser  pg-meta  dbuser_prometheus
bin/createdb    pg-meta  prometheus

Check database availability and create extensions。

psql postgres://dbuser_prometheus:[email protected]:5432/prometheus -c 'CREATE EXTENSION timescaledb;'

Configure Promscale

Install promscale by executing the following command on the meta node

yum install -y promscale

If not available in the default package, you can directly download it.

wget https://github.com/timescale/promscale/releases/download/0.6.1/promscale_0.6.1_Linux_x86_64.rpm
sudo rpm -ivh promscale_0.6.1_Linux_x86_64.rpm

Edit the promscale config file /etc/sysconfig/promscale.conf.

PROMSCALE_DB_HOST="127.0.0.1"
PROMSCALE_DB_NAME="prometheus"
PROMSCALE_DB_PASSWORD="DBUser.Prometheus"
PROMSCALE_DB_PORT="5432"
PROMSCALE_DB_SSL_MODE="disable"
PROMSCALE_DB_USER="dbuser_prometheus"

Finally start promscale, which will access the database instance with timescaledb installed and create the required schema.

# launch
cat /usr/lib/systemd/system/promscale.service
systemctl start promscale && systemctl status promscale

Configure Prometheus

Prometheus can use Remote Write/ Remote Read via Promscale, using Postgres as remote storage.

Edit Prometheus config file.

vi /etc/prometheus/prometheus.yml

Add the following record.

remote_write:
  - url: "http://127.0.0.1:9201/write"
remote_read:
  - url: "http://127.0.0.1:9201/read"

After restarting Prometheus, the monitoring data can be placed in Postgres.

systemctl restart prometheus