Pigsty v1.5.1 Documentation on PIGSTY

Introduction

Mon, 01 Jan 0001 00:00:00 +0000

Different users have different concerns. If you encounter problems, you are welcome to check the FAQ, submit an Issue, or ask the Community for help.

Beginners

Beginners can visit the Pigsty demo site: http://demo.pigsty.cc for a quick glance.

There are several data applications that are built upon Pigsty to illustrate this distribution: pglog, covid, isd, dbeng, worktime.

You can also deploy SaaS software with docker and get production-grade durability with an external pigsty database.

Developer

Developers are more concerned about the fastest way to download, install and access the database, please refer to Installation.

Installation

Mon, 01 Jan 0001 00:00:00 +0000

It takes 3 steps to install Pigsty: Prepare, Configure, Execute Playbook

There are two typical modes: Singleton & Cluster Management

Singleton Meta: Install pigsty on one single meta node. (for PostgreSQL dev/test, RDS monitoring, etc…)
Cluster Manage: Initiate control from meta node, and manage & monitoring more nodes & databases.

Singleton Meta

When Pigsty is installed on one single node. It will init pigsty infra with a battery including Postgres as CMDB. You may use that database directly.

Features

Mon, 01 Jan 0001 00:00:00 +0000

The battery-include, auto-piloting, handy & thrifty distribution for open-source databases.

Battery-Include Open-Source Postgres Distribution
Auto-Piloting monitoring and management SRE Solution
Easy-to-use Database-as-Code Developer Toolbox
Safe & Thrifty open-source alternative to Cloud RDS/PaaS

High Availability / Ultimate Observability / Handy Toolbox / Database as Code / Versatile Scenario / Safety & Thrifty

PostgreSQL Distribution

RedHat for Linux!

Pigsty deeply integrates the PostgreSQL 14.4 with powerful extensions: TimescaleDB 2.7, PostGIS 3.2, and Citus 11.0, and hundreds+ of extensions, all Battery-include.

FAQ: Frequently Asked Questions

Mon, 01 Jan 0001 00:00:00 +0000

Here are some frequently asked questions. If you have some unlisted questions, Contact Us, or submit an Issue.

Preparation

It would help if you ensured that the machine node hardware specifications and OS meet the installation requirements, see: preparation.

Node Requirement

Warning

At least 1Core/2GB is required from singleton meta installation. An x86_64 Processor is needed. ARM is not supported yet.

Concepts

Mon, 01 Jan 0001 00:00:00 +0000

Index

Module	Concept	Deploy	Config	Playbook
`INFRA`	Concept: INFRA	Deploy: INFRA	Config: INFRA	Playbook: INFRA
`NODES`	Concept: NODES	Deploy: NODES	Config: NODES	Playbook: NODES
`PGSQL`	Concept: PGSQL	Deploy: PGSQL	Config: PGSQL	Playbook: PGSQL
`REDIS`	Concept: REDIS	Deploy: REDIS	Config: REDIS	Playbook: REDIS

Glossary

Node

Meta

Source Package

Offline Software Package

Sandbox

Production Environment/Prod Env

Singleton Meta

Cluster Manage

Database Cluster

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty consists of several modules that can be combined according to different scenarios.

Module

Pigsty currently offers four modules.

INFRA is Pigsty’s infrastructure module, including monitoring/alerting/visualization/logging/DNS/NTP components.
NODES is the node management module for configuring nodes, installing software, and collecting metrics and logs.
PGSQL is a PostgreSQL deployment management module, including various PG cluster deployments and monitoring.
REDIS is the Redis deployment management module, including Redis standalone/native cluster/sentinel deployment and monitoring.

Module	Conecpt	Deploy	Config	Playbook
`INFRA`	Concept: INFRA	Deploy: INFRA	Config: INFRA	Playbook: INFRA
`NODES`	Concept: NODES	Deploy: NODES	Config: NODES	Playbook: NODES
`PGSQL`	Concept: PGSQL	Deploy: PGSQL	Config: PGSQL	Playbook: PGSQL
`REDIS`	Concept: REDIS	Deploy: REDIS	Config: REDIS	Playbook: REDIS

Usage

You can choose which modules to enable on which nodes based on different requirements.

Infrastructure

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty provides a complete, battery-included PaaS infra.

Pigsty, as a battery-included PaaS solution, will integrate the basic infra into the provisioning solution.

Overview

Pigsty deploys a complete set of infra on meta nodes (the currently installed node), including:

Component	Port	Default Domain	Description
Nginx	80	`pigsty`	Web Service Portal
Yum Repo	80	`yum.pigsty`	LocalYum Repo
Grafana	3000	`g.pigsty`	Monitoring System/Visualization Platform
AlertManager	9093	`a.pigsty`	Alert aggregation management component
Prometheus	9090	`p.pigsty`	Monitoring Time Series Database
Loki	3100	`l.pigsty`	Log Collection
Consul	8500	`c.pigsty`	Distributed Configuration Management and Service Discovery
Docker	2375	-	Container Platform
PostgreSQL	5432	-	Pigsty CMDB
Ansible	-	-	Initiate management commands
Consul DNS	8600	-	DNS Service（Optional）
Dnsmasq	53	-	DNS Server（Optional）
NTP	123	-	NTP Time Server（Optional）

A set of environments contains one or more meta nodes for infra deployment. All infrastructure components are deployed replica-style, except for DCS.

Concept: Nodes

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty use nodes for deployment, nodes cloud be physical machines, VMs, or even Pods.

Pigsty has two types of nodes: meta node and (normal) node.

Meta nodes are used to initiate control, and (normal) nodes are managed under control.

Meta node: Run infra.yml playbook to install Pigsty, INFRA, NODES, and PGSQL modules.
Node: Run nodes.yml playbook to join Pigsty, and install the NODES module by default.

Meta Node

Meta-Nodes are nodes installed with Pigsty, with admin capability and a complete infra set.

Concept: PGSQL

Mon, 01 Jan 0001 00:00:00 +0000

This article introduces the core concepts required for PostgreSQL cluster management.

PGSQL Cluster / ER Model / Identity Parameter
Cluster / Instance / Node / Service
PostgreSQL HA
Deploy: PGSQL ｜Config: PGSQL | Playbook: PGSQL ｜ Custom: PGSQL
PGSQL Service and PGSQL Access
PGSQL Privilege and PGSQL Authentication
PGSQL Users and PGSQL Database

PGSQL Cluster

PGSQL for production environments is organized in clusters, which clusters are logical entities consisting of a set of database instances associated by primary-replica. Each database cluster is a self-organizing business service unit consisting of at least one database instance.

Concept: Redis

Mon, 01 Jan 0001 00:00:00 +0000

This article introduces the core concepts required for Redis cluster management.

Deploy: Redis ｜Config: Redis | Playbook: Redis

ER Model

The Redis entity concept model is almost identical to PostgreSQL and includes the Cluster and Instance. Note that Cluster here does not refer to the clusters in Redis’ native clusters.

The core difference is that Redis typically uses multiple singleton instances, with multiple Redis instances typically deployed on a single physical/VM to take advantage of multi-core CPUs.

PGSQL Service & Access

Mon, 01 Jan 0001 00:00:00 +0000

Defining PostgreSQL service and achieving stable, reliable, and HA access through LB and connection pooling.

The concepts of Service and Access are proposed for HA PostgreSQL clusters.

Personal User

After completing the singleton deployment, port 5432 of this node provides PostgreSQL services, and port 80 provides UI class services.

On the current meta node, executing psql with no parameters using the admin user can connect directly to the local pre-defined meta database.

PGSQL Authentication and Privilege

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL provides a standard access control mechanism: Authentication and Privileges, both of which are based on the Role system.

Role

Pigsty’s default role system contains four default roles and four default users：

name	attr	roles	desc
dbrole_readonly	Cannot login		role for global readonly access
dbrole_readwrite	Cannot login	dbrole_readonly	role for global read-write access
dbrole_offline	Cannot login		role for restricted read-only access (offline instance)
dbrole_admin	Cannot login Bypass RLS	pg_monitor pg_signal_backend dbrole_readwrite	role for object creation
postgres	Superuser Create role Create DB Replication Bypass RLS		system superuser
replicator	Replication Bypass RLS	pg_monitor dbrole_readonly	system replicator
dbuser_monitor	16 connections	pg_monitor dbrole_readonly	system monitor user
dbuser_dba	Bypass RLS Superuser	dbrole_admin	system admin user

Default Roles

Pigsty has four default roles：

PGSQL Authentication and Privilege

Mon, 01 Jan 0001 00:00:00 +0000

PostgreSQL provides a standard access control mechanism: Authentication and Privileges, both of which are based on the Role system.

Role

Pigsty’s default role system contains four default roles and four default users：

name	attr	roles	desc
dbrole_readonly	Cannot login		role for global readonly access
dbrole_readwrite	Cannot login	dbrole_readonly	role for global read-write access
dbrole_offline	Cannot login		role for restricted read-only access (offline instance)
dbrole_admin	Cannot login Bypass RLS	pg_monitor pg_signal_backend dbrole_readwrite	role for object creation
postgres	Superuser Create role Create DB Replication Bypass RLS		system superuser
replicator	Replication Bypass RLS	pg_monitor dbrole_readonly	system replicator
dbuser_monitor	16 connections	pg_monitor dbrole_readonly	system monitor user
dbuser_dba	Bypass RLS Superuser	dbrole_admin	system admin user

Default Roles

Pigsty has four default roles：

Pigsty Deployment

Mon, 01 Jan 0001 00:00:00 +0000

It takes three steps to deploy Pigsty： Prepare, Configure, Playbook.

Preparation

Before installing Pigsty, you need to prepare the required resources: physical/VM nodes, admin users, and download Pigsty software.

Configuration

After preparation, you need to indicate to Pigsty what infra and database services you need via configure.

Playbook Execution

The next step can be to land the requirements by executing the playbook.

Preparation

Mon, 01 Jan 0001 00:00:00 +0000

How to prepare the resources required for Pigsty deployment.

Node Provisioning
Meta Provisioning
Admin Provisioning
Software Provisioning
- Pigsty source code
- Pigsty offline package
- Vagrant (sandbox)
- Virtualbox (sandbox)

Node Provisioning

Before deploying Pigsty, the user needs to prepare machine node resources with arbitrary database nodes, including at least one meta node.

The nodes can use any nodes: bare metals, local VMs, cloud VMs, containers, etc… only if the following conditions are met:

Sandbox

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty supports both local sandbox and cloud sandbox for quickly preparing a standard 1/4 node demo environment locally or in the cloud.

Pigsty provides a sandbox environment. Ultimately creating and running through the sandbox installation and deployment process for deployment in production envs have Pigsty very helpful.

Introduction

The config specifications, object identities, and default database predetermined environment of the sandbox are consistent in both the local and cloud versions.

Monitoring System

Mon, 01 Jan 0001 00:00:00 +0000

How to use Pigsty to monitor existing PostgreSQL instances?

For existing Pigsty instances that Pigsty does not create, some additional configuration is required if they are monitored using Pigsty’s monitoring system.

TL; DR

Create the monitoring object in the target instance: monitoring object configuration.

Declare the cluster in the inventory.

pg-test:
  hosts:                                # Assign unique local ports to each instance
    10.10.10.11: { pg_seq: 1, pg_role: primary , pg_exporter_port: 20001}
    10.10.10.12: { pg_seq: 2, pg_role: replica , pg_exporter_port: 20002}
    10.10.10.13: { pg_seq: 3, pg_role: offline , pg_exporter_port: 20003}
  vars:
    pg_cluster: pg-test                 # Fill in the cluster name
    pg_version: 14                      # Fill in the major version of the database
    pg_databases: [{ name: test }]      # Fill in the database list (each database object as an array element)

#  Provide monitoring user passwords in global/cluster/instance configs  pg_monitor_username/pg_monitor_password

Execute the playbook against the cluster: ./pgsql-monly.yml -l pg-test.

PostgreSQL Deployment

Mon, 01 Jan 0001 00:00:00 +0000

This article describes several ways to deploy a PostgreSQL cluster using Pigsty: PGSQL-related playbook and config. Please refer to the related doc.

Identity Parameters: Introduces the identity parameters required to define a standard PostgreSQL HA cluster.
Singleton Deployment: Defines a single instance PostgreSQL cluster.
Primary-Replica Cluster: Defines a standard availability cluster with one primary & one replica.
Sync-Standby: Define a highly consistent cluster with sync standby and RPO = 0.
Quorum Commit: Defines a cluster with higher data consistency: most replicas return commits on the successful side.
Offline Replica: Dedicated instances for hosting OLAP analysis, ETL, and interactive personal queries individually.
Standby Cluster: Produces real-time online clones of existing clusters for offsite disaster recovery or delayed.
Delayed Cluster: For responding to software/human failures such as mistaken table and database deletion, faster than PITR.
Cascade Instance: Used to build cascade within a cluster for many replica scenarios (20+) to reduce primary pressure.
Citus Deployment: Deploy Citus distributed database cluster.
MatrixDB Deployment: Deploy Greenplum7/PostgreSQL12 compatible chronological data warehouse.

Identity

The Core Identity Parameters are information that must be provided when defining a PostgreSQL cluster.

Redis Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty is a PostgreSQL distribution and a general-purpose application runtime. It can manage, deploy, and monitor other applications and databases, such as Redis.

Similar to PostgreSQL, deploying Redis requires the same two steps.

Declare/Define the Redis cluster
Execute Playbook to create the Redis cluster

Define Redis Cluster

ER Model

The Redis entity concept model is almost identical to PostgreSQL and includes the Cluster and Instance. Note that Cluster here does not refer to the clusters in Redis’ native clusters.

MatrixDB Deployment

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty can be used to deploy and monitor MatrixDB (equal to Greenplum 7+ Time-Series database).

Currently, MatrixDB uses PostgreSQL 12 kernel, while native Greenplum uses 9.6 kernels, so MatrixDB is used instead of Greenplum implementation, and Greenplum support will be added later.

E-R Model

MatrixDB is logically divided into two parts, Master and Segments, composed of PostgreSQL instances, divided into four categories: Master/Standby/Primary/Mirror.

Master is the port directly accessed by the user to take over queries. There is only one MatrixDB deployment, usually deployed using a standalone node.
Standby is a physical replica of the Master instance, which takes over when the Master fails and is usually deployed using a standalone node, which is optional.
A MatrixDB deployment typically has multiple Segments, each of which consists of a mandatory primary instance and an optional mirror instance.
The segment’s primary is responsible for the actual storage and computation, and the mirror does not carry the read and write traffic. It takes over for the primary when the primary is down, usually distributed on different nodes from the primary.
MatrixDB installation wizard determines the distribution of primary and mirror of Segment, and there may be different Segment instances on Segments nodes of the cluster.

Deployment conventions

Playbooks

Mon, 01 Jan 0001 00:00:00 +0000

Learn about the pre-built playbooks provided by Pigsty, the features, how to use them, and the considerations.

Pigsty implements core control functions at the bottom through the Ansible Playbook, and Pigsty provides pre-built playbooks in four main categories:

infra: Use the infra series of playbooks to install Pigsty singleton on the meta node with optional features.
nodes: Use the nodes series of playbooks to include more nodes in Pigsty monitoring and management and for subsequent use.
pgsql: Use the pgsql series of playbooks to deploy and manage PostgreSQL database clusters on existing nodes.
redis: Use the redis series of playbooks to deploy and manage various modes of Redis clusters on existing nodes.

Overview

Playbook	Function	Link
infra	Full installation of Pigsty on the meta node	`src`
`infra-demo`	Special playbook for complete initialization of a four-node demo sandbox in one go	`src`
`infra-jupyter`	Adding the optional data analysis service component Jupyter Lab to the meta node	`src`
nodes	Node provisioning to include nodes in Pigsty for subsequent database deployment	`src`
`nodes-remove`	Node remove, unloading node DCS and monitoring, no longer included in Pigsty	`src`
pgsql	PostgreSQL cluster deploy, or expand	`src`
`pgsql-remove`	PostgreSQL cluster destruction, or downsize	`src`
`pgsql-createuser`	Creating PostgreSQL business users	`src`
`pgsql-createdb`	Creating a PostgreSQL Business Database	`src`
`pgsql-monly`	Monly mode, with access to existing PostgreSQL instances or RDS	`src`
`pgsql-migration`	Generate PostgreSQL semi-automatic database migration solution (Beta)	`src`
`pgsql-matrixdb`	Reuse the PG role to deploy a MatrixDB data warehouse clusters (Beta)	`src`
redis	Deploy a Redis database in cluster/standalone/Sentinel mode	`src`
`redis-remove`	Redis cluster/node destruction	`src`

The typical use process is as follows：

Playbook: INFRA

Mon, 01 Jan 0001 00:00:00 +0000

Install pigsty & Add new optional to meta node with INFRA series playbooks.

Playbook	Function	Link
`infra`	Install pigsty on meta nodes	`src`
`infra-demo`	Special version of `infra`, pull up the 4-node sandbox in one-pass	`src`
`infra-remove`	Uninstall pigsty from the meta node.	`src`
`infra-jupyter`	Install optional JupyterLab on meta node	`src`

`infra`

The infra.yml playbook will install Pigsty on the meta nodes.

When using Pigsty as a battery-included Postgres, infra.yml is sufficient for the installation.

Playbook: NODES

Mon, 01 Jan 0001 00:00:00 +0000

Use the NODES playbook to bring more nodes to Pigsty, adjusting nodes to the state described in the config.

Once pigsty is installed on the meta node with infra.yml, You can add more nodes to Pigsty with nodes.yml or remove them from Pigsty with nodes-remove.yml.

Playbook	Function	Link
`nodes`	Node Provisioning. Register node into Pigsty and prepare for database deployment	`src`
`nodes-remove`	Node Removal, uninstall DCS & Monitoring & Logging, de-register from Pigsty	`src`

`nodes`

The nodes.yml playbook will register nodes to Pigsty.

Playbook: PGSQL

Mon, 01 Jan 0001 00:00:00 +0000

Pull up a defined cluster of HA PostgreSQL cluster using the PGSQL series playbook.

Overview

Playbook	Function	Link
`pgsql`	Deploy a PostgreSQL cluster, or cluster expand	`src`
`pgsql-remove`	Destroy PostgreSQL cluster, or cluster downsize	`src`
`pgsql-createuser`	Create PostgreSQL business users	`src`
`pgsql-createdb`	Create a PostgreSQL Business Database	`src`
`pgsql-monly`	Monly mode, with access to existing PostgreSQL instances or RDS	`src`
`pgsql-migration`	Generate PostgreSQL semi-automatic database migration solution (Beta)	`src`
`pgsql-matrixdb`	Reuse the PG to deploy a set of MatrixDB clusters (Beta)	`src`

`pgsql`

After completing the infra initialization, users can use pgsql.yml to complete the initialization of the database cluster.

Playbook: REDIS

Mon, 01 Jan 0001 00:00:00 +0000

REDIS series Playbook: Define and pull Redis databases in traditional standalone, native, and sentinel clusters.

Playbook	Function	Link
`redis`	Deploying a Redis database in Native/Standalone/Sentinel cluster	`src`
`redis-remove`	Redis cluster/node destruction	`src`

`redis`

Deploy redis instances on nodes.

# init all redis instances on group <cluster>
 ./redis.yml -l <cluster>       # create redis among group <cluster>

# init all redis instances specific node
 ./redis.yml -l 10.10.10.10     # setup all redis instances on node 10.10.10.10

# init one specific instance 10.10.10.11:6501 (skip redis node)
 ./redis.yml -l 10.10.10.11 -e redis_port=6501 -t redis

Alias script bin/createredis wrap above playbook with:

Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty uses declarative configuration.

Pigsty defines infra and database clusters through Inventory, and each Pigsty deploy has a corresponding config. Pigsty’s config uses the “Infra as Data” philosophy: Users describe requirements through the declarative config, and Pigsty adapts the fundamental components to the expected state.

Formally, the inventory can be implemented as a default local config file or as dynamic configuration data in CMDB. This article uses the default YAML configuration file pigsty.yml as an example. Pigsty detects the current node environment and generates the recommended config file in configure.

Config: Infra

Mon, 01 Jan 0001 00:00:00 +0000

Use the INFRA Playbook, and deploy the PGSQL cluster to adjust the cluster state to the state described in PGSQL Config.

Use the INFRA series playbooks to configure the Pigsty infra.

Infra config deals with such issues: localYum repos, machine node base services: DNS, NTP, kernel modules, parameter tuning, admin users, installing packages, DCS Server setup, monitor infra installation, and initialization (Grafana, Prometheus, Alertmanager), global traffic portal Nginx config, etc.

Usually, the infra requires very few modifications, and the main modification is just a text replacement of the meta node IPs, which is done in ./configure automatically. The other occasional change is to the access domain defined in nginx_upstream. Other parameters are adjusted as needed.

Config: Nodes

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty provides host provisioning and monitoring functions. The nodes.yml playbook can be executed to configure the node to the corresponding state and incorporate it into the Pigsty monitor system.

NODE_IDENTITY : Node identity parameters
NODE_DNS: Node domain name resolution
NODE_PACKAGE: Node Repo & Packages
NODE_TUNE: Node features & kernel tuning
NODE_ADMIN: Node admin user & dir
NODE_TIME: Node time zone, NTP, crontab
DOCKER: Node docker daemon
NODE_EXPORTER: Node metrics exporter
PROMTAIL: Logger agent

ID	Name	Section	Type	Level	Comment
300	`meta_node`	`NODE_IDENTITY`	bool	C	mark this node as meta
301	`nodename`	`NODE_IDENTITY`	string	I	node instance identity
302	`node_cluster`	`NODE_IDENTITY`	string	C	node cluster identity
303	`nodename_overwrite`	`NODE_IDENTITY`	bool	C	overwrite hostname with nodename
304	`nodename_exchange`	`NODE_IDENTITY`	bool	C	exchange static hostname
310	`node_etc_hosts_default`	`NODE_DNS`	string[]	C	static DNS records
311	`node_etc_hosts`	`NODE_DNS`	string[]	C/I	extra static DNS records
312	`node_dns_method`	`NODE_DNS`	enum	C	how to setup dns service?
313	`node_dns_servers`	`NODE_DNS`	string[]	C	dynamic DNS servers
314	`node_dns_options`	`NODE_DNS`	string[]	C	/etc/resolv.conf options
320	`node_repo_method`	`NODE_PA`	enum	C	how to use yum repo (local)
321	`node_repo_remove`	`NODE_REPO`	bool	C	remove existing repo file?
322	`node_repo_local_urls`	`NODE_REPO`	url[]	C	local yum repo url list
330	`node_packages_default`	`NODE_PACKAGES`	string[]	C	pkgs to be installed on all node
331	`node_packages`	`NODE_PACKAGES`	string[]	C	extra pkgs to be installed
332	`node_packages_meta`	`NODE_PACKAGES`	string[]	G	meta node only packages
333	`node_packages_meta_pip`	`NODE_PACKAGES`	string	G	meta node pip3 packages
340	`node_disable_firewall`	`NODE_TUNE`	bool	C	disable firewall?
341	`node_disable_selinux`	`NODE_TUNE`	bool	C	disable selinux?
342	`node_disable_numa`	`NODE_TUNE`	bool	C	disable numa?
343	`node_disable_swap`	`NODE_TUNE`	bool	C	disable swap?
344	`node_static_network`	`NODE_TUNE`	bool	C	use static DNS config?
345	`node_disk_prefetch`	`NODE_TUNE`	bool	C	enable disk prefetch?
346	`node_kernel_modules`	`NODE_TUNE`	string[]	C	kernel modules to be installed
347	`node_tune`	`NODE_TUNE`	enum	C	node tune mode
348	`node_sysctl_params`	`NODE_TUNE`	dict	C	extra kernel parameters
350	`node_data_dir`	`NODE_ADMIN`	path	C	default data disk mountpoint
351	`node_admin_enabled`	`NODE_ADMIN`	bool	G	create admin user?
352	`node_admin_uid`	`NODE_ADMIN`	int	G	admin user UID
353	`node_admin_username`	`NODE_ADMIN`	string	G	admin user name
354	`node_admin_ssh_exchange`	`NODE_ADMIN`	bool	C	exchange admin ssh keys?
355	`node_admin_pk_current`	`NODE_ADMIN`	bool	A	pks to be added to admin
356	`node_admin_pk_list`	`NODE_ADMIN`	key[]	C	add current user’s pkey?
360	`node_timezone`	`NODE_TIME`	string	C	node timezone
361	`node_ntp_enabled`	`NODE_TIME`	bool	C	setup ntp on node?
362	`node_ntp_service`	`NODE_TIME`	enum	C	ntp mode: ntp or chrony?
363	`node_ntp_servers`	`NODE_TIME`	string[]	C	ntp server list
364	`node_crontab_overwrite`	`NODE_TIME`	bool	C/I	overwrite instead of append /etc/crontab
365	`node_crontab`	`NODE_TIME`	string[]	C/I	crontab list of node
370	`docker_enabled`	`DOCKER`	bool	C	docker enabled?
371	`docker_cgroups_driver`	`DOCKER`	string	C	docker cgroup driver
372	`docker_registry_mirrors`	`DOCKER`	string[]	C	docker registry mirror location
373	`docker_image_cache`	`DOCKER`	path	C	docker image cache tarball
380	`node_exporter_enabled`	`NODE_EXPORTER`	bool	C	node_exporter enabled?
381	`node_exporter_port`	`NODE_EXPORTER`	int	C	node_exporter listen port
382	`node_exporter_options`	`NODE_EXPORTER`	string	C/I	node_exporter extra cli args
390	`promtail_enabled`	`PROMTAIL`	bool	C	promtail enabled ?
391	`promtail_clean`	`PROMTAIL`	bool	C/A	remove promtail status file ?
392	`promtail_port`	`PROMTAIL`	int	G	promtail listen port
393	`promtail_options`	`PROMTAIL`	string	C/I	promtail cli args
394	`promtail_positions`	`PROMTAIL`	string	C	path to store promtail status file

`NODE_IDENTITY`

Each node has identity parameters that are configured through the parameters in <cluster>.hosts and <cluster>.vars.

Config: PGSQL

Mon, 01 Jan 0001 00:00:00 +0000

Use the PGSQL Playbook, and deploy the PGSQL cluster to adjust the cluster state to the state described in the PGSQL config.

Pigsty has 100+ config parameters for describing a PostgreSQL cluster. However, users usually only need to care about a few parameters in identity params and business objects: the former expresses the database cluster “Who is it? Where is it?” and the latter represents the database “What does it look like? What’s in it?”.

Config: REDIS

Mon, 01 Jan 0001 00:00:00 +0000

Config Redis cluster and manipulate REDIS playbook behavior. Refer to Redis Deployment for details.

REDIS_IDENTITY: REDIS Identity Params
REDIS_NODE: REDIS Software, Dir & Exporter
REDIS_PROVISION: REDIS Server Provisioning

ID	Name	Section	Type	Level	Comment
700	`redis_cluster`	`REDIS_IDENTITY`	string	C	redis cluster identity
701	`redis_node`	`REDIS_IDENTITY`	int	I	redis node identity
702	`redis_instances`	`REDIS_IDENTITY`	instance[]	I	redis instances definition on this node
710	`redis_fs_main`	`REDIS_NODE`	path	C	main data disk for redis
711	`redis_exporter_enabled`	`REDIS_NODE`	bool	C	install redis exporter on redis nodes
712	`redis_exporter_port`	`REDIS_NODE`	int	C	default port for redis exporter
713	`redis_exporter_options`	`REDIS_NODE`	string	C/I	default cli args for redis exporter
720	`redis_safeguard`	`REDIS_PROVISION`	bool	C	set to true to disable purge
721	`redis_clean`	`REDIS_PROVISION`	bool	C	purge existing redis during init
722	`redis_rmdata`	`REDIS_PROVISION`	bool	C	remove redis data dir with it?
723	`redis_mode`	`REDIS_PROVISION`	enum	C	standalone,cluster,sentinel
724	`redis_conf`	`REDIS_PROVISION`	string	C	which config template will be used
725	`redis_bind_address`	`REDIS_PROVISION`	ip	C	e.g 0.0.0.0, empty will use inventory_hostname as bind address
726	`redis_max_memory`	`REDIS_PROVISION`	size	C/I	max memory used by each redis instance
727	`redis_mem_policy`	`REDIS_PROVISION`	enum	C	memory eviction policy
728	`redis_password`	`REDIS_PROVISION`	string	C	empty password disable password auth (masterauth & requirepass)
729	`redis_rdb_save`	`REDIS_PROVISION`	string[]	C	RDB save cmd, disable with empty array
730	`redis_aof_enabled`	`REDIS_PROVISION`	bool	C	enable redis AOF
731	`redis_rename_commands`	`REDIS_PROVISION`	object	C	rename dangerous commands
732	`redis_cluster_replicas`	`REDIS_PROVISION`	int	C	how much replicas per master in redis cluster ?

`REDIS_IDENTITY`

Identity parameters are the information that must be provided to define a Redis cluster, including:

Customize: PGSQL

Mon, 01 Jan 0001 00:00:00 +0000

The Patroni template is used to customize the specification config of the PostgreSQL cluster, while the Postgres template is used to customize the content of the PostgreSQL cluster.

Pigsty provides nearly 100 parameters on PGSQL describing the PostgreSQL cluster.

However, if you profoundly customize the database cluster created by Pigsty, you can see the Patroni template and Postgres template.

Patroni Templates

Pigsty uses Patroni to manage and initialize Postgres clusters. Suppose you wish to modify the default config params, specifications and tuning schemes, HA policies, DCS access, and control APIs of the PostgreSQL cluster. You can do so by modifying the Patroni template.

Pigsty Dashboards

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty provides a professional and easy-to-use PostgreSQL monitor system that distills the industry’s monitoring best practices.

The Pigsty monitoring dashboard consists of several relatively independent boards.

Application	Description
Home	Home
`PGSQL`	PostgreSQL Database Monitor
`REDIS`	Redis Database Monitor
`NODES`	Host Node Monitor
`INFRA`	Infra monitoring/logging
APP	Extra added applications

HOME

Pigsty’s home page provides navigation to the various boards.

HOME

PGSQL

The PostgreSQL monitoring dashboard has its level, from the top down, as follows：

Service Discovery

Mon, 01 Jan 0001 00:00:00 +0000

This article describes the mechanism used by the Pigsty monitor system Prometheus to discover monitor objects.

The basis of service discovery is identity. For more identity information, please refer to the section entity.

It would help if you also associated monitor targets with the identity in the monitor system, and Pigsty provides two implementations.

Static File Service Discovery: Using an automatically maintained configuration file (default).
Consul service discovery: Uses automatically maintained Consul service registration information.

The static file is the default service discovery mechanism. Before v1.0.0, Consul was the default service discovery method, and the discovery mechanism could be configured via parameters.

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

Metric is the core concept of Pigsty’s monitor system.

Format

Metrics are formally cumulative, atomic logical units of measure that can be updated and statistically aggregated over periods.

Metrics typically exist as time series with dimension labels. For example, pg:ins:qps_realtime in the Pigsty sandbox refers to the presentation of real-time QPS for all instances.

pg:ins:xact_commit_rate1m{cls="pg-meta", ins="pg-meta-1", ip="10.10.10.10", role="primary"} 0
pg:ins:xact_commit_rate1m{cls="pg-test", ins="pg-test-1", ip="10.10.10.11", role="primary"} 327.6
pg:ins:xact_commit_rate1m{cls="pg-test", ins="pg-test-2", ip="10.10.10.12", role="replica"} 517.0
pg:ins:xact_commit_rate1m{cls="pg-test", ins="pg-test-3", ip="10.10.10.13", role="replica"} 0

Users can perform operations on indicators: summation, derivation, aggregation, etc.

Alerting

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty uses Prometheus as the primary alerting system.

Prometheus + AlertManager (primary)
Grafana (backup), not enabled by default

Alert

Alerts are critical for daily fault response and improving system availability.

Missing alarms will lead to reduced availability, and false alarms will lead to reduced sensitivity, and it is necessary to design the alarm rules prudently.

A reasonable definition of alert levels and the corresponding processing flow.
A reasonable definition of alert indicators, removal of duplicate alert items, and replenishment of missing alert items.
Scientific config of alert thresholds based on historical monitoring data reduces the false alert rate.
Reasonably rationalize the special case rules to eliminate false alerts caused by maintenance work, ETL, and offline queries.

Alert Taxonomy

Categorized by source module

Install Applications

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty can be used to deploy and monitor PostgreSQL and make and distribute data Applications.

Pigsty provides three sample applications.

pglog, which analyzes PostgreSQL CSV log samples.
covid, which visualizes WHO COVID-19 data and accesses country outbreak data.
pglog, NOAA ISD, allows querying weather observation records from 1901 for 30,000 surface weather stations.

Structure of the application

A Pigsty application typically includes at least one or all of the following.

Docker Applications

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty v1.5.1 comes with Docker and Docker Compose deployment support, where Docker Daemon will be enabled by default on the meta node.

You can use Docker to deploy and launch software applications quickly. You can directly access the PostgreSQL/Redis database deployed on the host in the container using the connection string.

PgAdmin4: A GUI tool for managing PostgreSQL instances.
PGWeb: A tool automatically generates back-end API services based on PG database schema.
PostgREST: A tool to automatically generate backend API services based on PG database schema.
ByteBase: A GUI tool for making PostgreSQL schema changes.
Jupyter Lab: A battery-included Python lab environment for data analysis and processing.

You can also use Docker to execute some battery-included command tools.

Upgrade Grafana Backend Database

Mon, 01 Jan 0001 00:00:00 +0000

You can use postgres as the database used by the Grafana backend.

In this tutorial, you will learn about the following.

How to create a new cluster
How to create a new biz user in an existing database cluster
How to create a new biz database in an existing database cluster
How to access databases created by Pigsty
How to manage dashboards in Grafana
How to manage PostgreSQL DataSources in Grafana
How to do upgrade the grafana database

TL; DR

vi pigsty.yml   # uncomment user/db definition：dbuser_grafana  grafana
bin/createuser  pg-meta  dbuser_grafana
bin/createdb    pg-meta  grafana

psql postgres://dbuser_grafana:DBUser.Grafana@meta:5436/grafana -c \
  'CREATE TABLE t(); DROP TABLE t;'    # check pgurl connectivity

vi /etc/grafana/grafana.ini            # edit [database] section: type & url
systemctl restart grafana-server

Create Postgres Cluster

We can define a new database grafana on pg-meta. A Grafana-specific database cluster can also be created on a new machine node: pg-grafana.

Jupyter Lab

Mon, 01 Jan 0001 00:00:00 +0000

TL;DR

./infra-jupyter.yml # install jupyter on meta node on port 8888 with user jupyter and password pigsty
./infra-jupyter.yml -e jupyter_port=8887 # use another port 8887 by default
./infra-jupyter.yml -e jupyter_username=osuser_jupyter jupyter_password=pigsty2 # use another password

Jupyter Config

ID	Name	Section	Type	Level	Comment
220	`jupyter_port`	`JUPYTER`	int	G	jupyter server listen port
221	`jupyter_username`	`JUPYTER`	bool	G	os user for jupyter lab
222	`jupyter_password`	`JUPYTER`	bool	G	password for jupyter lab

Jupyter Lab is a complete data science R&D env based on IPython Notebook for data analysis and visualization. It is currently an optional Beta feature and is only enabled in the demo by default.

Backup and Recovery

Mon, 01 Jan 0001 00:00:00 +0000

Backup & Recovery matters.

Failures can be divided into two categories: hardware/resource failures and software/human errors.

Replication aims for the former, while delayed replica & cold backups aim for the latter.

Pigsty has complete support for backups. There is battery-included physical replication that can be configured by simply declaring replica & delayed roles. And handy cold backup scripts for cold backup.

Physical Replica

In Pigsty, physical backups are created by specifying roles (pg_role) for the database instances. For example, the following configuration declares a HA database cluster with one primary & two replicas.

Offline Installation

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty downloads all dependent packages from the Internet and creates local Yum repo during initialization.

The total size of all the dependent software is about 1GB or so. Although Pigsty has tried to use mirror repos as much as possible to speed up the download, the download of a small number of packages may still be blocked by firewalls and may appear very slow. You can set the download proxy to complete the first download by using the proxy_env config entry.

CMDB Usage

Mon, 01 Jan 0001 00:00:00 +0000

Instead of a static config file, you can use postgres as an inventory for Pigsty.

Using CMDB as a dynamic Inventory for Ansible has several advantages: metadata is presented as data tables in a highly structured way, and database constraints ensure consistency. The CMDB also allows you to use third-party tools to manage Pigsty metadata.

Currently, Pigsty’s CMDB only supports PostgreSQL clusters. If your pigsty.yml contains Redis and MatrixDB, it will report an error. It is recommended to use a separate pigsty.yml config file to manage Redis and Greenplum clusters.

Migration Tutorial

Mon, 01 Jan 0001 00:00:00 +0000

An auxiliary playbook, pgsql-migration.yml, provides a battery-included migration method based on logical replication.

By filling in the information about the source and host clusters, the playbook will automatically create the scripts needed for the migration and simply execute them in sequence during the database migration.

activate                        # activate migration context
check-replica-identity          # prepare: make sure all table have replica identity
check-replica-identity-solution # prepare: fix table without replica identity
check-special-objec             # prepare: check special object: matrialized view
compare            # compare: fast check on data consistency (by row count)
copy-schema        # migration: copy schema from src to dst cluster
create-pub         # migration: create publication on source cluster
create-sub         # migration: build logical replication between src & dst clusters
progress           # migration: print logical replication progress
copy-seq           # migration: copy sequence number from src to dst cluster
next-seq           # migration: advance dst cluster by 10000 to fix primary confliction
remove-sub         # remove subscription from dst cluster

Prepare

SRC and DST Clusters

Suppose you want to migrate the pg-meta cluster in the sandbox (containing the Pigsty meta DB with the pgbench test tables) to the pg-test cluster.

PGSQL SOP: Standard Operating Procedure

Mon, 01 Jan 0001 00:00:00 +0000

Most cluster management operations require using the admin user on the meta node and the execution of the corresponding Ansible Playbook in the Pigsty root dir.

Unless otherwise specified, the following examples demonstrate a three-node cluster pg-test in a sandbox.

Cheatsheet

Cluster Admin

Manage PostgreSQL clusters and instances by executing the following commands on the meta node using the admin user.

FHS: File Hierarchy Structure

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty FHS

#------------------------------------------------------------------------------
# pigsty
#  ^-----@app                    # extra demo application resources
#  ^-----@bin                    # bin scripts
#  ^-----@docs                   # document (can be docsified)
#  ^-----@files                  # ansible file resources
#            ^-----@conf         # config template files
#            ^-----@rule         # soft link to prometheus rules
#            ^-----@ui           # soft link to grafana dashboards
#  ^-----@roles                  # ansible business logic
#  ^-----@templates              # ansible templates
#  ^-----@vagrant                # sandbox resources
#  ^-----configure               # configure wizard script
#  ^-----ansible.cfg             # default ansible config file
#  ^-----pigsty.yml              # default config file
#  ^-----*.yml                   # ansible playbooks

#------------------------------------------------------------------------------
# /etc/pigsty/
#  ^-----@targets                # file based service discovery targets definition
#  ^-----@dashboards             # static grafana dashboards
#  ^-----@datasources            # static grafana datasources
#  ^-----@playbooks              # extra ansible playbooks
#------------------------------------------------------------------------------

Prometheus FHS

#------------------------------------------------------------------------------
# Config FHS
#------------------------------------------------------------------------------
# /etc/prometheus/
#  ^-----prometheus.yml              # prometheus main config file
#  ^-----alertmanager.yml            # alertmanger main config file
#  ^-----@bin                        # util scripts: check,reload,status,new
#  ^-----@rules                      # record & alerting rules definition
#            ^-----@infra            # infrastructure rules & alert
#            ^-----@nodes            # nodes rules & alert
#            ^-----@pgsql            # pgsql rules & alert
#            ^-----@redis            # redis rules & alert
#            ^-----@..........       # etc...
#  ^-----@targets                    # file based service discovery targets definition
#            ^-----@infra            # infra static targets definition
#            ^-----@nodes            # nodes static targets definition
#            ^-----@pgsql            # pgsql static targets definition
#            ^-----@redis            # redis static targets definition
#            ^-----@.....            # other targets
#------------------------------------------------------------------------------

Postgres FHS

The following parameters are related to the PostgreSQL database dir:

HA Scenarios

Mon, 01 Jan 0001 00:00:00 +0000

You can strengthen your confidence in the cluster’s HA capability through a HA scenario experiment.

Below is a list of 24 typical HA failure scenarios, divided into three categories: Primary failure, Replica failure, and DCS failure, with eight specific scenarios in each category.

All experiments assume that HA auto-switchover mode is enabled, where Patroni should correctly handle Primary and Replica failures.

Number	Case Name	Auto Mode	Manual switch
A	Primary Node Failures
1A	Primary Node node down	Failover	Manual switch
2A	Primary Node Postgres process shutdown (`pg_ctl or kill -9`)	Failover	Manual reboot
3A	The Primary Node Patroni process is shut down usually (`systemctl stop patroni`)	Failover	Manual reboot
4A	Abnormal shutdown of the Primary Node Patroni process (`kill -9`)	Needs confirmation	No effect
5A	Primary Node load hit full, false death (watchdog)	Needs confirmation	No effect
6A	Primary DCS Agent is not available (`systemctl stop consul`)	Cluster Primary demotion	No effect
7A	Primary Node network jitter	Automatic Failover on timeout	Need to observe
8A	Erroneous deletion of Primary data dir	Automatic Failover	Manual switch
B	Replica bank failure (1/n , n>1)
1B	Replica Node down	No effect	No effect
2B	Replica Node Postgres process shutdown (`pg_ctl or kill -9`)	No effect	No effect
3B	Replica Node process Postgres Manual Shutdown (`pg_ctl`)	No effect	No effect
4B	Replica Node Patroni process exception Kill (`kill -9`)	No effect	No effect
5B	Replica DCS Agent is not available (`systemctl stop consul`)	No effect	No effect
6B	Replica Node Load hit full, false death	Depends	Depends
7B	Replica Node network jitter	No effect	No effect
8B	Boosting a Replica node by mistake (`pg_ctl promte`)	Automatic recovery	Split Brain
C	DCS failure
1C	DCS Server is completely unavailable (most nodes are unavailable)	Downgrade all cluster Primary nodes	No effect
2C	DCS pass Primary, not Replica (1 Primary & 1 Replica)	No effect	No effect
3C	DCS pass Primary, not Replica (1 Primary n Replica, n>1)	No effect	No effect
4C	DCS pass Replica, not Primary (1 Primary, 1 Replica)	No effect	No effect
5C	DCS pass Replica, not Primary (1 Primary n Replica, n>1)	Automatic Failover	No effect
6C	DCS network jitter: simultaneous outages, Primary and Replica nodes recover simultaneously, or the Primary node recovers first	No effect	No effect
7C	DCS network jitter: simultaneous outages, Replica nodes recover first, Primary nodes recover later (1 Primary, 1 Replica)	No effect	No effect
8C	DCS network jitter: simultaneous interruptions, Replica nodes recover first, Primary nodes recover later (1 Primary n Replica, n>1)	Automatic Failover over TTL	No effect

Environment

The following is a walkthrough of a local Pigsty four-node sandbox.

Common Failures

Mon, 01 Jan 0001 00:00:00 +0000

Hardware Faults

ID	Name	Symptom	Process
H01	Primary node down	pg_up = 0 for 1-3 minutes	No immediate intervention required. Additional examples after the fact Removal from Access Domain Execute Case 8: Cluster Role Adjustment
H02	Replica node down	pg_up = 0 for 1-3 minutes	No immediate intervention required Adding examples after the fact. Removal from Access Domain Execute Case 8: Cluster Role Adjustment
H03	Primary node network partition	Loss of all monitoring data of the primary instance, network unreachable	Confirm Failover status Force Fencing old primary if necessary
H04	Replica Node Network Partitioning	Loss of all monitoring data from the instance, network unreachable	Usually no effect, waiting for recovery Contact O&M and network engineers to handle
H05	TCP the retransmission rate is too high	TCP Retrans stay high for a long time, a lot of Conn Reset, a lot of query requests fail	Find O&M and network engineers to handle
H06	Node memory error	EDAC counter growth, system error log	After confirming that there are no errors in the replica memory Execute Case 10: cluster primary-replica switch
H07	Bad blocks on disk, data corruption	Query results, and logs show error messages such as can’t read block	Execute Case 10: cluster primary-replica switch Manual data recovery using data recovery tools
R01	High CPU usage	CPU / load / pressure index high	topCheck for large CPU footprint programs and clean them up As in the case of an avalanche, execute a kill query stop.
R02	OOM appears	Process Failure appears, OOM message, high memory usage, start using SWAP	Confirm memory, confirm SWAP topCheck for large memory hogs and clean them up Re-pulling the killed process Emergency SWAP partition addition
R03	Disk Full	Disk Write Full Database Crash A large number of shell commands cannot be executed	Remove `/pg/dummy` to free up emergency space Check and handle WAL buildup Check aa and process a large number of Log files Confirm whether the business has cleanable data
R06	Disk/network card IO too high	Disk/NIC BandWidth too large Disk > 2GB/s Network > 1 GB/s	Check applications that use the network/disk, such as backups, to add speed limits.

Software Errors

ID	Name	Symptom	Process
SP1	Database process abort	`ps aux` can’t find the postgres process	Check Postgres, Patroni status Confirm Failover results, or perform Failover manually
SP2	Connection pool process aborted	`systemctl status pgbouncer` Failure	restart service component or reset service component
SP3	Primary Patroni process aborted	`systemctl status patroni` Failure	As above, enter maintenance mode, reboot or reset Patroni
SP4	Primary Consul process aborted	`systemctl status consul` Failure	As above, enter maintenance mode, reboot or reset Consul
S05	HAProxy process aborts	`systemctl status haproxy` Failure	As above, restart or reset Haproxy
S06	Connection pool contamination	An error message similar to Cannot execute XXX on read-only transactions appears	Restart the Pgbouncer connection pool or configure `server_reset_query`
S07	Connection pool cannot connect to the database	pgbouncer can not connect to server	Check whether the user, password, and HBA configuration are correct Execute Case-4: Cluster Service User Creation to refresh the user
S08	Connection pool reaches QPS bottleneck	PGbouncer QPS reaches 3 to 4W, CPU usage reaches 100%	Use multiple Pgbouncers (not recommended) Use Default service to bypass Pgbouncer Notify business side of speed limit
S09	DCS Server is not available	In auto-switchover mode, all primary will go to the unwritable state after TTL	Set all clusters to maintenance mode immediately
S10	DCS Agavailableavailable	If it is a replica, it has no effect, if it is a primary, it will be demoted to a replica and the cluster is not writable	Set all clusters to maintenance mode immediately
S11	XID Wraparound	Enter protection mode when age remaining 1000w.	This problem should be avoided in advance through monitoring locate the over-aged databases and tables, perform emergency cleaning quickly locate the cause of blocking the vacuum and solve restore in single user mode
S12	WAL Stacking	WAL size continues to grow	Execute `CHECKPOINT`multiple times confirm the wal archive status confirm whether there are unfinished ultra-longg transactions from the replica confirm whether there are replication slots to prevent wal recycling

Human Errors

ID	Name	Symptom	Process
M01	Mistakenly deleted database clusters	The database cluster is gone	Use cold standby to recover the cluster Prepare to run
M02	Mistakenly elevating an instance to the primary	split-brain	No need to handle it in automatic mode, otherwise, split-brain
M03	Erased data by mistake	The data is gone	Stop vacuum, use `PG_ Dirtyread` extract extract from delayed cluster extract and restore from cold standby
M04	Erasure Form	The table is gone	Fetch from delayed cluster Fetch and restore from cold standby
M05	Integer Sequence Number Overflow	Sequence exceeds INTMAX	Refer to integer primary key online upgrade manual to handle
M06	Insert data conflicts due to duplicate primary key serial numbers	violate constratint …	Grow serial number value (e.g. +100000)
M07	Slow query queuing / avalanche	Large number of slow query logs	Use pg_terminate_backend to periodically clean up slow queries (e.g. every 1 second)
M08	Deadlock queuing / avalanche	Lock stacking	Use pg_terminate_backend to periodically clean up queries (e.g. every 1 second)
M09	HBA denied access	no HBA entry for xxx	Case 6: APPLY-PGSQL-HBA
M10	User password error	password auth failure for xxx	Case 4: Create OGSQL Biz User
M11	Insufficient access privileges	permission denied for x	Check if the user created the object with the correct admin Refer to Default Privilege to manually fix the object privileges

Community

Mon, 01 Jan 0001 00:00:00 +0000

There’s an active Pigsty community. Search WeChat pigsty-cc and join the user group.

Most answers cloud be found in FAQ. If there’s no answer to your questions, you can ask in the community.

It’s most welcome to create Issues or Pull Requests if you find some repeatable bugs.

Following information cloud helps a lot when asking for help:

What commands are you typing?
What result do you expect?
What environment is used?
- Is it a CentOS 7.8 fresh node?
- Is it a physical/virtual machine, or a public cloud ECS?
- Is anything else running on that node?
- Is offline packagepkg.tgz been used?
  - If yes, in /tmp/pkg.tgz, right?
  - If not, do you have proper Internet access?
- Is there any special restriction ? (such as firewall, ssh ban, etc…)

Mail

Author: Vonng (rh@vonng.com)

Github Issues

https://github.com/Vonng/pigsty/issues

https://t.me/joinchat/gV9zfZraNPM3YjFh

Discord

https://discord.gg/wDzt5VyWEz

Roadmap

Mon, 01 Jan 0001 00:00:00 +0000

History

Time	Description	Release
2019-05-15	POC	fa2ade31 (v0.0.1)
2020-04-30	First Commit	dd64677 (v0.0.2)
2020-06-20	Validation on Testing Environment	v0.1.0
2020-06-22	Interface enhancement	v0.0.3
2020-07-10	PGSQL Monitoring v6 GA	v0.2.0
2020-07-27	Refactor playbooks into ansible roles	v0.0.4
2020-08-19	Offline Installation Mode	v0.0.5
2020-10-22	Provisioning Solution GA	v0.3.0
2020-12-14	PostgreSQL 13 Support, Official Documentation	v0.4.0
2021-01-07	Databaes Customize Template	v0.5.0
2021-02-19	Architecture Enhancement	v0.6.0
2021-03-01	Monitor only deployment	v0.7.0
2021-03-28	Service Provision	v0.8.0
2021-04-04	Pigsty GUI, CLI, Logging Intergration	v0.9.0
2021-04-20	Accessibility and extensibility enhancement	v0.9.1
2021-07-26	v1 GA, Monitoring System Overhaul	v1.0.0
2021-10-12	HomePage, JupyterLab, Pgweb, Pev2 & Pgbadger	v1.1.0
2021-11-03	Upgrade default Postgres to 14, monitoring existing pg	v1.2.0
2021-11-30	PGCAT Overhaul & PGSQL Enhancement & Redis Support Beta	v1.3.0
2022-03-31	MatrixDB Support, Separated INFRA, NODES, PGSQL, REDIS	v1.4.0
2022-04-20	Bug fix & Full translation of English documents.	v1.4.1
2022-05-31	Docker Applications	v1.5.0

Timeline

Development Log

Mon, 01 Jan 0001 00:00:00 +0000

2022-06-18

1.5.1 for PostgreSQL 14.4
Upgrade haproxy to 2.6.0
repotrack i686 issus on el7
New apps: wiki.js ferretDB

2022-05-21

Grafana Logo Customize
Nginx Log Rework

2022-05-20

Prometheus Overview polish
Grafana Overview polish
DCS Overview polish

2022-05-15

new role: etcd
monitoring for etcd
Make etcd a viable option for ha PG DCS
Infra Dashboards rework
fix nodes/infra/pgsql config docs
Update docker containers, pgadmin4 to 6.9

2022-05-14

fix redis join issue
optimize consul role implementation

2022-05-13

pg_exporter metrics & dashboard calibration

2022-05-12

upgrade bytebase to v1.0.5

2022-05-09

v1.5.0-alpha.1 released
repo upstream calibration
build docker.tgz image cache and add to download script

2022-05-07

grant backup related permissions to pg_replication_username
add pg_probackup to default packages

2022-05-06

Docker application calibration

2022-05-05

New CMDB Design
Now cmdb works for redis & gpsql

2022-05-04

Dashboard Overhaul: Infra-Overview
Add node_crontab_overwrite implementation
Test citus/dcs3/mxdb conf
node_local_repo_url -> node_repo_local_urls

2022-05-02

add crontab implementation

2022-05-01

Refactor redis.yml & redis-remove.yml
Remove role redis, redis-remove, redis-exporter
Fix dashboard redis aliveness issue
Upgrade to redis v6.2.7

2022-04-30

Register Infra as common consul service
Make consul service discovery great again
implement node_data_dir which create /data during node init if not exists
add pg_probackup to default package list
integration test on pigsty-citus

2022-04-29

New Vars:

Why Pigsty

Mon, 01 Jan 0001 00:00:00 +0000

Why use Pigsty?

Our philosophy is: use the good database, use the good database so that there is no hard-to-use database in the world!

The database is the core component of an information system, a relational database is the absolute mainstay of the database, and PostgreSQL is the world’s most advanced open-source relational database.

PG provides a perfect enough database kernel, but it is not so simple to really use it well, and we help users to do that.

User Interface

Mon, 01 Jan 0001 00:00:00 +0000

User Interface

After completing the installation, the GUI provided by Pigsty can be accessed through a browser.

http://g.pigsty -> http://10.10.10.10:80 (nginx) -> http://10.10.10.10:3000 (grafana)

Visit http://<node_ip>:3000 to browse Pigsty home (username: admin, password: pigsty)

You can visit http://demo.pigsty.cc to view the public Pigsty Demo and browse through the features provided by the Pigsty Monitoring System.

Web Services

Pigsty will provide services to the public through a series of ports, and web services will be accessed uniformly through Nginx port 80.

Deploy

Mon, 01 Jan 0001 00:00:00 +0000

There are 3 steps to deploy pigsty: prepare, configure, playbook

Prepare

Configure

Playbook

INFRA Init
PGSQL initialization (cluster creation, new instances)
PGSQL Remove (Remove instance, remove cluster)
Create Biz user
Create Biz database

Customization of Postgres Template

Mon, 01 Jan 0001 00:00:00 +0000

Parameters

The template database template1 in a cluster can be customized using the PG template configuration item.

This ensures that any database newly created in this database cluster comes with the same default configuration: schema, extensions, and default permissions.

name	type	hierarchy	description
pg_init	`string`	G/C	Custom PG initialization scripts
pg_replication_username	`string`	G	PG replication user
pg_replication_password	`string`	G	PG replication user’s password
pg_monitor_username	`string`	G	PG monitor user
pg_monitor_password	`string`	G	PG monitor user password
pg_admin_username	`string`	G	PG Admin User
pg_admin_password	`string`	G	PG admin user password
pg_default_roles	`role[]`	G	Default created roles and users
pg_default_privileges	`string[]`	G	Database default permissions configuration
pg_default_schemas	`string[]`	G	The schema created by default
pg_default_extensions	`extension[]`	G	Default installed extensions
pg_offline_query	`bool`	I	Whether to allow offline queries
pg_reload	`bool`	A	Whether to reload database configuration (HBA)
pg_hba_rules	`rule[]`	G	Global HBA rules
pg_hba_rules_extra	`rule[]`	C/I	Cluster/instance specific HBA rules
pgbouncer_hba_rules	`rule[]`	G/C	Pgbouncer global HBA rules
pgbouncer_hba_rules_extra	`rule[]`	G/C	Pgbounce specific HBA rules
pg_databases	`database[]`	G/C	business database definitions
pg_users	`user[]`	G/C	business user definitions

When customizing the database template, the relevant parameters are first rendered as SQL scripts and then executed on the deployed database cluster.

Redis Deploy & Monitor

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty is also a universal application runtime that could be used for deploying & monitoring other databases and applications, such as Redis.

It takes two steps to deploy a Redis cluster:

Declare it
Execute playbook

Define Redis

Redis Config Entries

E-R Model

The Redis entity conceptual model is almost identical to [PostgreSQL]c-pgsql.md#ER-Model), and also includes the concepts of Cluster and Instance. Note that the concept of Cluster here does not refer to the clusters in Redis’ native clustering scheme.

PGWeb

Mon, 01 Jan 0001 00:00:00 +0000

Pigsty v1.5.1 does not ship the former infra-pgweb.yml playbook or the pgweb_enabled / pgweb_username inventory parameters. PGWeb is provided as a Docker application at app/pgweb.

Run PGWeb

Docker is enabled on the meta node in the bundled v1.5.1 inventory. Start the tagged PGWeb application with:

cd ~/pigsty/app/pgweb
docker-compose up -d

The tagged Compose file maps host port 8886 to container port 8081. Supply a database connection string in the PGWeb interface, for example postgres://dbuser_dba:DBUser.DBA@127.0.0.1:5432/meta after replacing the credentials as appropriate.

Using TimescaleDB to store Prometheus data

Mon, 01 Jan 0001 00:00:00 +0000

You can use postgres as the remote storage database used by the Prometheus backend.

While this is not the recommended behavior, it is a good opportunity to understand how the Pigsty deployment system is used.

Preparing the Postgres database

vi pigsty.yml # Uncomment DB/User definition: dbuser_prometheus prometheus

pg_databases:                           # define business users/roles on this cluster, array of user definition
  - { name: prometheus, owner: dbuser_prometheus , revokeconn: true, comment: prometheus primary database }
pg_users:                           # define business users/roles on this cluster, array of user definition
  - {name: dbuser_prometheus , password: DBUser.Prometheus ,pgbouncer: true , createrole: true,  roles: [dbrole_admin], comment: admin user for prometheus database }

Create a Prometheus business database with business users.