tag:blogger.com,1999:blog-8269109576314793902024-03-19T01:48:34.181-07:00Evol MonkeyA PostgreSQL blogEvol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.comBlogger44125tag:blogger.com,1999:blog-826910957631479390.post-1679003255891773672019-10-15T14:52:00.000-07:002019-10-15T14:52:15.208-07:00Tuning checkpointsWe recently had the chance to help a customer with some IO related issues that ended up being unconfigured checkpoints. Something that may not always be obvious but can actually be somewhat common.<br />
<br />
Let's start with how things roughly work.<br />
Postgres smallest IO unit is a disk block that is 8kb (by default). Each time postgres needs a new block it will fetch it from the disks and load it to an area in RAM called shared_buffers.<br />
When postgres needs to write, it does it in the same manner:<br />
<ul>
<li>Fetches the block(s) from the disks if the block is not in shared_buffers</li>
<li>Changes the page in shared buffers.</li>
<li>Marks the page as changed (dirty) in shared buffers.</li>
<li>It writes the change in a "sequential ledger of changes" called WAL to ensure durability.</li>
</ul>
<br />
This basically means that the writes are not yet "on disk". This operation is taken care of by a postgres process called checkpointer. Checkpoints are how postgres guarantees that data files and index files will be updated with all the changes that happened before that checkpoint. In case of a crash, postgres will go back to the latest checkpoint record and it will start a REDO operation from WAL. Checkpoints are triggered every <b>checkpoint_timeout</b> (default at 5min) or when changes reach <b>max_wal_size</b> (default at 1GB). This is an IO intensive operation and postgres tries to spread this IO<br />
with <b>checkpoint_completion_target</b> (default at 0.5).<br />
<br />
<b>checkpoint_timeout*</b> is the maximum time between checkpoints in seconds.<br />
<b>min_wal_size</b> minimum size of wals that will be recycled rather than removed<br />
<b>max_wal_size**</b> maximum size allowed for wals between checkpoints<br />
<b>checkpoint_completion_target </b>allows data changes to spread over a longer period of time, making the final fsync() much cheaper.<br />
* Affects recovery time, change only after reviewing the documentation<br />
** This is a softmax, it can exceed this value in special cases.<br />
<br />
Best way to start is to set checkpoint_timeout value to something reasonable and set max_wal_size high enough so you won't reach the timeout. To make sense of what is reasonable, you can do the following: schedule (cronjob will do) something like this to run in short periods of time, say every minute:<br />
<br />
psql -XqtA -c "copy (select now()::timestamp(0), pg_current_wal_insert_lsn()) to stdout with csv;" monkey >> current_lcn<br />
<br />
Leave it running for as long as you see fit, from the result you can extract the difference of 2 locations in bytes like this:<br />
<br />
monkey=# SELECT pg_size_pretty(pg_wal_lsn_diff('0/B1277248','0/59CEA2F8'));<br />
pg_size_pretty<br />
----------------<br />
1398 MB<br />
(1 row)<br />
<br />
(protip: file_fdw + window function)<br />
<br />
This function will calculate the difference of later location - earlier location in bytes, so having the location per minute can help calculate (or graph) the rate of changes over time.<br />
With that number, your storage capabilities and your recovery needs in mind you should be able to come up with a good starting point.<br />
<b>max_wal_size</b> should be set high enough so it won't be reached before the timeout. The rate of changes figure we calculated earlier should be a good indication of where to start. <b>min_wal_size</b> has to follow common sense and leave a small portion of wal files to be recycled for the next checkpoint.<br />
<b>checkpoint_completion_target</b> often is just set to 0.9 but if you want to be more "precise" it follows the following rule:<br />
<b>(checkpoint_timeout - 2min) / checkpoint_timeout</b><br />
<br />
To watch how checkpointer is working after the changes, you can query <a href="https://www.postgresql.org/docs/current/monitoring-stats.html#PG-STAT-BGWRITER-VIEW">pg_stat_bgwriter</a>. This is a very interesting view that probably deserves it's own blogpost - because of the data you can extract from it - but today we are going to concentrate on 2 columns, checkpoint_timed and checkpoint_req.<br />
<br />
<b>checkpoint_timed</b> counts checkpoints that happened triggered by checkpoint_timeout and <b>checkpoint_req</b> is the count of the checkpoint that happened because <b>max_wal_size</b> was hit.<br />
What you want to see there is the majority of the checkpoints to be timed.<br />
<br />
And because graphs are cool, you can see an example of a production system before and after checkpoint configuration. The graph is write IOPS over time on an AWS RDS database.<br />
In RDS specifically, i find it weird that checkpoint defaults are so low considering the high shared_buffers default it has.<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTmw4EzLSZarumOAU0UV_Yo-qwMqwwQDVTL06MyzdVgQ9DLT1JR4lzVQ_lFrIUBCsJWE7nf__Z7cuYr9M-nAx9TMGCqt9X0I9kxM7_FuQJzOJkhiYGNPLl5EhT-NyVk_2jZMI14UH7gwxI/s1600/Screen+Shot+2019-10-15+at+17.36.50.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="282" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTmw4EzLSZarumOAU0UV_Yo-qwMqwwQDVTL06MyzdVgQ9DLT1JR4lzVQ_lFrIUBCsJWE7nf__Z7cuYr9M-nAx9TMGCqt9X0I9kxM7_FuQJzOJkhiYGNPLl5EhT-NyVk_2jZMI14UH7gwxI/s640/Screen+Shot+2019-10-15+at+17.36.50.png" width="640" /></a></div>
<br />
<br />
<br />
<table class="crayon-table" style="background-image: none !important; border-collapse: collapse !important; border-spacing: 0px !important; border: none !important; caret-color: rgb(86, 100, 111); color: #56646f; font-family: Consolas, ConsolasRegular, "Courier New", monospace; font-size: 12px; margin-bottom: 0px !important; margin-left: -19px; margin-right: 0px !important; margin-top: 0px !important; max-width: 100%; outline: 0px; padding: 0px !important; table-layout: auto !important; vertical-align: baseline; width: auto !important;"><tbody style="border: 0px; font-style: inherit; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"></tbody></table>
Thanks for reading<br />
-- Vasilis Ventirozos<br />
-- credativ LLC<br />
<br />
<br />
<br />Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com32tag:blogger.com,1999:blog-826910957631479390.post-83213033662599361112019-07-11T06:14:00.000-07:002019-07-11T06:14:03.225-07:00Comparing pg 9.4 with pg 12, CTE editionPostgres 12 is around the corner, and as always is packed with new features. One of them being the option to allow the user to control the behaviour of CTE materialization. You can find the commit from Tom Lane <a href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=608b167f9f9c4553c35bb1ec0eab9ddae643989b">here</a>, which explains everything about this new change but TLDR is that so far, CTE's where fully materialized, so restrictions from the query that uses it won't apply to the CTE. Which is the right way doing this when you are using CTEs to INSERT/UPDATE/DELETE or when they are recursive. This means that when a CTE is side-effect-free and non-recursive it's safe to push the restrictions from the outer query.<br />
So, from postgres 12, when it's safe or when the CTE is called only once, postgres will inline the CTE to the outer query, removing the optimization fence. User will be able to override this behaviour by using MATERIALIZED / NOT MATERIALIZED keywords.<br />
<br />
Here's an example :<br />
<pre><code class="sql">
drop table if exists test_cte;
create table test_cte as
select generate_series(1,1000000) as id,
floor(random() * 10 + 1)::int as random;
-- OLD WAY
EXPLAIN analyze
WITH a_cte AS MATERIALIZED
(
SELECT random,count(*) FROM test_cte group by random
)
SELECT * FROM a_cte WHERE random = 5;
-- NEW
EXPLAIN analyze
WITH a_cte AS NOT MATERIALIZED
(
SELECT random,count(*) FROM test_cte group by random
)
SELECT * FROM a_cte WHERE random = 5;
</code></pre>
<br />
It's not really important to show all the differences in explain plans but rather to see how many rows it had to process in order to create the CTE by applying the condition directly:<br />
<pre><code class="sql">
-- Materialized (old):
-> Parallel Seq Scan on test_cte (cost=0.00..8591.88 rows=416688 width=4) (actual time=0.031..19.077 rows=333333 loops=3)
-- Not Materialized (New):
-> Parallel Seq Scan on test_cte (cost=0.00..9633.59 rows=2083 width=4) (actual time=0.021..24.469 rows=33222 loops=3)
Filter: (random = 5)
Rows Removed by Filter: 300112
</code></pre>
<br />
A production system running a 2TB on 9.4 with a dev that has a lot of free diskspace that we plan upgrading soon makes a perfect candidate to test. Since 9.4 is the one going EOL and 12 being the latest, we could make an interesting performance comparisons.<br />
<br />
Some details about this comparison :<br />
I've used pg_stat_statements to extract from production 10 statements that are a combination of the most slow and most often used ones. <u>I didn't filter out statements based on if the new behaviour will apply or not. So this comparison, while directed towards CTE's, it's not strictly about it.</u> It mostly shows how long has postgres has come over the last years. The comparison run on 2 freshly analyzed copies of production, located in the same filesystem using common hardware.<br />
<br />
Here's the results :<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRcFylVg3eELEAitIrRX3aymiz8UizMZM3FZZtongfReZIzYLwaqBvWF8wDPFECGjMlSLSNHPy_gLZ98x7G7qOwIB8phy1I6Q1NEa57_sJK9rBLiO3R5OtEH7REbyZEMX8ZKhelpOpxGud/s1600/20190711124645_hd.png" imageanchor="1"><img border="0" height="470" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhRcFylVg3eELEAitIrRX3aymiz8UizMZM3FZZtongfReZIzYLwaqBvWF8wDPFECGjMlSLSNHPy_gLZ98x7G7qOwIB8phy1I6Q1NEa57_sJK9rBLiO3R5OtEH7REbyZEMX8ZKhelpOpxGud/s640/20190711124645_hd.png" width="640" /></a><br />
<br />
<br />
<br />
As you can see, when postgres 12 is faster the difference is substantial, after manually examining statements 4,6 and 7<br />
I verified that it was because of the materialization. Interestingly, even by forcing the materialization, postgres 12 was still faster, unfortunately i didn't keep these numbers.<br />
<br />
This feature is obviously welcome, and even if changing the default behaviour might sound a little scary, the benefits are just too big to ignore. That said, i think it would be great if this behaviour could be manipulated in session instead of each individual statement.<br />
<br />
<br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">Thanks for reading</span><br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">Vasilis Ventirozos</span><br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617"><a href="https://www.credativ.com/">https://www.credativ.com</a></span><br />
<br />
<br />Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com4tag:blogger.com,1999:blog-826910957631479390.post-6777488078934912382019-06-28T12:45:00.000-07:002019-06-29T06:53:08.095-07:00Accessing PostgreSQL data from AWS Lambda<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">All databases need some maintenance that we usually perform using scheduled tasks. For example, if you have an RDS instance and you want to get a bloat report once a month, you’ll probably need a small EC2 instance just to do these kinds of things. In this post, we will talk about accessing RDS, getting a result from a SQL statement and reporting it to a slack channel using a scheduled Lambda function; let’s call this poor man’s crontab :)</span></div>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">Before we start, psycopg2 isn’t supported by lambda so it needs to be packaged into the lambda deployment package along with any other dependencies, but since psycopg2 requires libpq it needs to be compiled with libpq statically linked. There are many binaries of psycopg2 that you can download, but I would suggest you compile your own, using the latest PostgreSQL source code. We’ll talk about all these steps throughout this post.</span></span><br />
<span style="font-family: "arial"; font-size: 11pt; white-space: pre-wrap;">We’ll cover the following:</span></div>
<ul><span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
<li><span style="font-family: "arial"; font-size: 11pt; white-space: pre-wrap;">Creating a new AWS postgres RDS instance</span></li>
<li><span style="font-family: "arial"; font-size: 11pt; white-space: pre-wrap;">How to compile psycopg (we’ll use docker for that)</span></li>
<li><span style="font-family: "arial"; font-size: 11pt; white-space: pre-wrap;">Code for the function</span></li>
<li><span style="font-family: "arial"; font-size: 11pt; white-space: pre-wrap;">Packaging</span></li>
<li><span style="font-family: "arial"; font-size: 11pt; white-space: pre-wrap;">Lambda function, trigger, schedule</span></li>
<li><span style="font-family: "arial"; font-size: 11pt; white-space: pre-wrap;">Testing</span></li>
</span></ul>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
</span>
<br />
<div dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;">
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">I won’t get into much detail about the steps of building an RDS instance as it is pretty straight forward, assuming you have your aws client setup, you’ll need something similar to the following, if not, go to aws console and clickety-click.</span></span></div>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
</span>
<br />
<pre><span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617"><code class="bash"> #!/bin/sh
aws rds create-db-instance \
--db-subnet-group-name [your subnet group] \
--db-security-groups [your security group] \
--db-instance-identifier [your instance identifier] \
--db-instance-class db.t2.micro \
--engine postgres \
--allocated-storage 5 \
--no-publicly-accessible \
--db-name [your db name] \
--master-username [your username] \
--master-user-password [your password]\
--backup-retention-period 3
</code></span></pre>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
For compiling psycopg2 we need a linux installation, I am on a mac (don’t hate) so I used a docker container (ubuntu bionic) for the compiles.
You can find the Dockerfile <a href="https://github.com/vventirozos/pg_lambda/blob/master/Dockerfile">here</a>. </span><br />
<br />
The steps for compiling are the following :
<br />
<pre><span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617"><code class="bash"># Download sources :
wget -c https://ftp.postgresql.org/pub/source/v11.4/postgresql-11.4.tar.gz
git clone https://github.com/psycopg/psycopg2
# Untar
tar zxfv postgresql-11.4.tar.gz
# Make a local install dir
mkdir ./pgsql-11.4
#configure, compile, install postgres
cd postgresql-11.4 ; ./configure --prefix=/home/postgres/pgsql-11.4 --without-readline --without-zlib
make ; make install
# Edit psycopg2 setup config
cd psycopg2 ;vim setup.cfg
# Change:
static_libpq=1
pg_config=/home/postgres/pgsql-11.4/bin/pg_config
# Compile psycopg2
python3.6 setup.py build
# Package psycopg2
mv ~/psycopg2 ~/psycopg2_source
cp -r ~/psycopg2_source/build/lib.linux-x86_64-3.6/psycopg2/ ~/
tar cvfz psycopg2.tgz psycopg2/
# Finally, get the package from docker, I just used scp
scp psycopg2.tgz vasilis@192.168.1.10:
</code></span></pre>
<br />
Next we need to write the Lambda function.<br />
<u>You can find my sample code, along with a binary version of psycopg2 for 11.4, and a deployment script in my repo <a href="https://github.com/vventirozos/pg_lambda">here</a>. </u><br />
<br />
Keep in mind that the filename (lambda_function.py) and the function name (lambda_handler) are important as they will be defined later when we will create the
function in lambda and that the handler has to have this syntax structure :
<br />
<pre><code class="python">def handler_name(event, context):
...
return some_value </code> </pre>
For more information see AWS documentation <a href="https://docs.aws.amazon.com/lambda/latest/dg/python-programming-model-handler-types.html">here</a>.
The code is pretty simple, it will connect to an RDS instance, it’ll run a simple statement and it will report the result in slack, but it could
be something different like aws SNS or your trigger event might be something completely different, like when a file is created in S3 do something
in python and load it to postgres, lambda also supports API gateway.
For more information about the different things you can do, see <a href="https://docs.aws.amazon.com/lambda/latest/dg/lambda-services.html#intro-core-components-event-sources">here</a>.<br />
<br />
Next, we need to package all these into a “lambda deployment package” AWS documentation can be found <a href="https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html">here</a>.
This is basically a zip that contains all dependencies and the code we want lambda to execute. In my case Slacker (and its dependencies) and the
psycopg2 package we compiled earlier. Amazons documentation is pretty clear how to do this, so I won’t repeat the steps here but your
zip file should look like this when unzipped:
<br />
<pre><code class="bash">
> tree -L 1
.
├── bin
├── certifi
├── certifi-2019.6.16.dist-info
├── chardet
├── chardet-3.0.4.dist-info
├── config.py
├── idna
├── idna-2.8.dist-info
├── lambda_function.py
├── psycopg2
├── requests
├── requests-2.22.0.dist-info
├── slacker
├── slacker-0.13.0.dist-info
├── urllib3
├── urllib3-1.25.3.dist-info
└── util.py
</code></pre>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
Now that we have the deployment package ready, all we need to do is the lambda function and something that will trigger it on schedule.
Go to aws console, hit lambda and create a new function:
</span>
<br />
<div>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617"><img height="680" src="https://lh5.googleusercontent.com/5KsKof2gu7u7Ji4xwhDwavkDNDQeOlpV3x3uRnsn2sCt3E1IgXO5zTKnrZu8GXzhCUi-heSRScm0t96vsbhIdqswUMtwz7GdujRf_eLIskrkRZfzdAJv4ankMxW6QPWcU2T7EPy9" style="border: none;" width="624" /></span></div>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
</span>
<br />
<div>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
</span>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">Next screen, on the left side where triggers are, pick cloudwatch events, we’ll use these events to schedule how often the function will run, we also should put python 3.6 in runtime and in Handler we make sure that it matches the filename.hundler_function_name of our code.
</span><br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
</span>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617"><img height="460" src="https://lh6.googleusercontent.com/1br2IoafuNWm7KKd4AlmVt9kGcfTx5FRdOhjImpY4tkQ5v_zdRyv3FcBA1UgUlTZ0b1618wFVv02mjn-cDfqcyWCjM8B1cKqXiNN4H952NBOMhy-ShNeBSbky2ExxaOOdmne5Pcf" style="border: none;" width="624" />
</span><br />
<br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
</span>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">Next screen you’ll have to create a new rule and write a schedule expression :
</span><br />
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"><br /></span>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
</span>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617"><span id="docs-internal-guid-28de3b66-7fff-57d3-5f30-92b05d27f064"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"><img height="472" src="https://lh5.googleusercontent.com/n6eShQDPZAbs1ELzszO6CXoyqBiFMtVsz56UzfFWiYyhWUEhXVxApHX_MUNlHnBm-I0i7RgwubRba10JLaRiwsUfdNd2q8XypMyPs2HiI0PeqmvgXJFCjD8fTSFDxUYpDMPV5xjQ" style="border: none;" width="624" /></span></span></span><br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">
</span>
<span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">
</span>
This schedule expression accepts ‘rate’ or a cron expression, see AWS documentation <a href="https://docs.aws.amazon.com/lambda/latest/dg/tutorial-scheduled-events-schedule-expressions.html">here</a>.</span><br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617"> From the function main page in aws console, scroll down to network, add your VPC and last, At ‘Execution role’ add a role or create a new one like the following :
</span><br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
</span>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617"><span id="docs-internal-guid-3c7adaec-7fff-7630-ce2f-6c06fa08fb3e"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"><img height="662" src="https://lh6.googleusercontent.com/wPvhB2bTqmTXGgDxDnusXr1bK55XnEz62WLmn0R_BwvwY7irefpVvMsaujD71Hdw_UEP93SulAeHI_oeYPpKD5HhAAOSb2y4mYupbUDVhwsdQ3otHJ2Bh9lPFX-7s6-g4hgkKV3T" style="border: none;" width="527" /></span></span></span><br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;">
</span>
As you may have noticed, I haven't added any code yet. You can add your zip from the console :
</span><br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
</span>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617"><span id="docs-internal-guid-f8e20c9e-7fff-3087-ed1c-1a8a91f4106f"><span style="font-family: "arial"; font-size: 11pt; vertical-align: baseline; white-space: pre-wrap;"><img height="177" src="https://lh4.googleusercontent.com/xY0ivPemxL6rZY_v5DzHytgUe1-fSrL6E7do3sZ-_mG0HGzX6-p3UObWDYlXv_EhXHH2YPmflCp1Nz-R6c6dWLh6cvlYDbaGALaDEwv3OKCUqFOAw3qnqPAmBu4-qszGBD1w-f2j" style="border: none;" width="624" /></span></span></span><br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">I found it much easier to use a script like the following in order to make testing a bit less tedious. Keep in mind that the following script requires AWS CLI installed:
</span><br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
</span>
<br />
<pre><span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617"><code class="bash">home=/Users/vasilis/Work/working_on/projects/lambda/packaging
cp ./function_work_in_progress.py ./lambda_function.py
rm -f package.zip
cd packages ; zip -r9 /Users/vasilis/Work/working_on/projects/lambda/packaging/function.zip .
cd $home
zip -g function.zip lambda_function.py
zip -g function.zip config.py
zip -g function.zip util.py
aws lambda update-function-code --function-name pg_lambda --zip-file fileb://function.zip
sleep 2
rm -f outfile ;aws lambda invoke --function-name "pg_lambda" outfile ;cat outfile</code></span></pre>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
</span>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">In this script, I repackage the deployment package, I update the function code on aws and last, I manually invoke the function, which should give me a result in slack. I found this way much easier to work with than making inline changes in AWS console. </span><br />
Lambda is cool.. kinda, and having a way to schedule simple postgres tasks without having a server, or an ec2 instance is nice, but this is by no means a replacement for crontab, simply because lambda has a lot of limitations. Say you wanted a pgbadger report sent everyday, you are out of luck. Perl is not supported in lambda. But if you want a bloat report or any SQL statement executed on a schedule, lambda could be a solution.<br />
<br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
</span>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">As mentioned before, dockerfile , scripts and a binary of psycopg2 (against pg11.4) can be found <a href="https://github.com/vventirozos/pg_lambda">here</a>.</span><br />
<br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
</span>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">Thanks for reading</span><br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">Vasilis Ventirozos</span><br />
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617"><a href="https://www.credativ.com/">https://www.credativ.com</a></span></div>
<span id="docs-internal-guid-43b63d00-7fff-2594-a722-e2f9e3e37617">
</span>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com10tag:blogger.com,1999:blog-826910957631479390.post-68058450557330586142019-06-21T10:01:00.001-07:002019-06-21T11:51:18.398-07:00CVE-2019-10164 Who's affected and how to protect your systems.<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Yesterday, 20th of June Postgres community released minor version updates for all supported versions. </span><span style="font-family: inherit;">(11.4, 10.9, 9.6.14, 9.5.18, 9.4.23, 12Beta2)</span><br />
<span style="font-family: inherit;">As with any minor version, it is recommended to upgrade and keep your database to the latest minor version. But this release is a bit more important </span><br />
<span style="font-family: inherit;">than others because it includes a fix about a recently </span><span style="font-family: inherit;">discovered CVE (CVE-2019-10164) </span><br />
<span style="font-family: inherit;">From the release notes:</span><br />
<span style="font-family: inherit;"><br /></span>
<i><span style="font-family: inherit;">"This release is made outside of the normal update release schedule as the security</span></i><br />
<i><span style="font-family: inherit;">vulnerability was determined to be critical enough to distribute the fix as quickly as possible.</span></i><br />
<i><span style="font-family: inherit;">Users who are running PostgreSQL 10, PostgreSQL 11, or the PostgreSQL 12 beta should upgrade as soon as possible."</span></i><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">In this post i'll talk about this CVE. What this all about, who is affected and how </span><span style="font-family: inherit;">to protect your systems against it.</span><br />
<span style="font-family: inherit;"><br /></span>
<br />
<h4>
<span style="font-family: inherit;">What is this all about.</span></h4>
<span style="font-family: inherit;">A system authenticated user could change their own password with a "special crafted password" that could crash your database server or, </span><br />
<span style="font-family: inherit;">worst case, have the operating system user who owns postgres executing arbitrary code. This is a bug on SCRAM authentication mechanism </span><br />
<span style="font-family: inherit;">which is why it only affects postgres version 10 and above. Reality is that on </span>some operating systems, for example ubuntu, the default compiler options<br />
should prevent code execution (see <a href="https://usn.ubuntu.com/4027-1/">here</a>) and since <span style="font-family: inherit;">an attack can only happen from a user that is already authenticated, this user, with enough </span>privileges<span style="font-family: inherit;"> </span><br />
<span style="font-family: inherit;">could potentially come up with an SQL statement that could DOS or even bring the service down</span><span style="font-family: inherit;">. </span><br />
<b><span style="font-family: inherit;">Of course, this doesn't mean that if you trust your users </span><span style="font-family: inherit;">you shouldn't upgrade, you definitely should.. </span></b><br />
<span style="font-family: inherit;">The latest pg versions also fix a second attack in libpq, a library that is extensively used by almost all common postgres clients</span><br />
<span style="font-family: inherit;">psql, psycopg2 and pgadmin4 included. In this attack a user connected from a server that runs software that uses libpq, </span><br />
<span style="font-family: inherit;">say psycopg2 or psql could crash the client or run arbitrary code by exploiting the SCRAM authentication process. </span><br />
<span style="font-family: inherit;"><b>In this case the exploitation will happen on the client, not the server.</b></span><br />
<span style="font-family: inherit;"><b><br /></b></span>
<span style="font-family: inherit;"><b><br /></b></span>
<br />
<h4>
<span style="font-family: inherit;">Upgrade.. do it!</span></h4>
<span style="font-family: inherit;">The solution is simple, </span>(always) <span style="font-family: inherit;">upgrade to the latest minor version, this CVE is just an extra reason to do so.</span><br />
<span style="font-family: inherit;"><b>Keep in mind that even if you don't use SCRAM, an authenticated user could still exploit this.</b></span><br />
<span style="font-family: inherit;">To prevent the client side attack <b>you should also upgrade your postgres clients</b> ensuring that you </span><span style="font-family: inherit;">have the latest libpq installed. </span><br />
<span style="font-family: inherit;">This could be tricky because it might include things that are easy to forget like for example using postgres from AWS lambda </span><br />
<span style="font-family: inherit;">where you need to compile psycopg2 with the libpq library statically linked. </span><br />
<span style="font-family: inherit;">In this case you would need to recompile psycopg2 and create a new deployment package.</span><br />
<span style="font-family: inherit;">The easiest way to see the libpq version is to do a psql --version as they have the same version. </span><br />
<span style="font-family: inherit;">If you have custom software compiled using libpq </span><span style="font-family: inherit;">you can get the version by invoking</span><br />
PQlibVersion() in a small program like this :<br />
<br />
#include <stdio.h><br />
#include <libpq-fe.h><br />
<br />
int main() {<br />
int lib_ver = PQlibVersion();<br />
printf("libpq version: %d\n", lib_ver);<br />
<br />
return 0;<br />
}<br />
<br />
The program above can be compiled like this :<br />
<i>gcc -o pq_version pq_version.c -I/opt/pgsql/pgsql-11.4/include/ -lpq -std=c99</i><br />
and it should generate an output that looks like this (for 11.4) :<br />
<b>libpq version: 110004</b><br />
<br />
Truth is that if you are using 10 and above, no matter which version you are running, you have to recompile using the new library.<br />
Keep in mind that all the above is for installations that have compiled from source, if you are using packages, your package management system<br />
should take care of everything, all you need to do is upgrade your packages to the latest version ; just be sure to do the upgrades on your client machines<br />
(like web servers or other such machines) as well as the database server this time.<br />
<span style="font-family: inherit;">And since i mentioned AWS, Postgres RDS is at 10.7 and 11.2 and Aurora at 10.7 as of today.</span><br />
<span style="font-family: inherit;">I assume that because of the CVE they should be releasing new versions soon, so upgrade the moment they announce something :)</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Thanks for reading,</span><br />
<span style="font-family: inherit;">Vasilis Ventirozos</span><br />
<span style="font-family: inherit;">www.credativ.com</span><br />
<span style="font-family: inherit;"><br /></span>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com1tag:blogger.com,1999:blog-826910957631479390.post-28774429716088118262019-02-06T12:07:00.003-08:002019-02-06T12:21:12.958-08:00Managing xid wraparound without looking like a (mail) chimp<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">My colleague Payal came across an outage that happened to mailchimp's mandrill app yesterday, link can be found <a href="https://www.mediapost.com/publications/article/331556/mailchimps-mandrill-app-suffers-service-outage-c.html">HERE</a>.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">Since this was PostgreSQL related i wanted to post about the technical aspect of it.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">According to the company :</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<i><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">“Mandrill uses a sharded Postgres setup as one of our main datastores,”</span></i><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">the email explains.</span><br />
<i><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">“On Sunday, February 3, at 10:30pm EST, 1 of our 5 physical Postgres instances saw a significant spike in writes.” </span></i><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">The email continues:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">“<i>The spike in writes triggered a Transaction ID Wraparound issue. When this occurs, database activity is completely halted. The database sets itself in read-only mode until offline maintenance (known as vacuuming) can occur.”</i></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">So, lets see what that "transaction id wraparound issue" is and how someone could prevent similar outages from ever happening.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">PostgreSQL uses MVCC to control transaction visibility, basically by comparing transaction IDs (XIDs). A row with an insert XID greater than the current transaction XID shouldn't be visible to the current transaction. But since transaction IDs are not unlimited a cluster will eventually run out after</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">(2^32 transactions 4+ billion) causing transaction ID wraparound: transaction counter wraps around to zero, and all past transaction would appear to be in the future</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">This is being taken care of by vacuum that will mark rows as frozen, indicating that they were inserted by a transaction that committed far in the past that can be visible to all current and future transactions. To control this behavior, postgres has a configurable called <b>autovacuum_freeze_max_age</b>, which defaults at 200.000.000 transactions, a very conservative default that must be tuned in larger production systems.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">It sounds complicated but its relatively easy not to get to that point,</span><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">for most people just having autovacuum on will prevent this situation from ever happening. You can simply schedule manual vacuums by getting a list of </span><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">the tables "closer" to autovacuum_freeze_max_age with a simple query like this:</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<br />
<pre style="background-color: #f0f0f0; border: 1px dashed rgb(204, 204, 204); color: black; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"> SELECT 'vacuum analyze ' || c.oid::regclass ||' /* manual_vacuum */ ;'
FROM pg_class c LEFT JOIN pg_class t ON c.reltoastrelid = t.oid
WHERE c.relkind = 'r'
ORDER BY greatest(age(c.relfrozenxid),age(t.relfrozenxid)) desc
LIMIT 10;
</span></code></pre>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">Even if you want to avoid manual vacuums, you can simply create a report or a monitoring metric based on age of relfrozenxid of pg_class combined with pg_settings, eg :</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<br />
<pre style="background-color: #f0f0f0; border: 1px dashed rgb(204, 204, 204); color: black; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> SELECT oid::regclass::text AS table,pg_size_pretty(pg_total_relation_size(oid)) AS table_size,age(relfrozenxid) AS xid_age,
(SELECT setting::int FROM pg_settings WHERE
name = 'autovacuum_freeze_max_age') -
age(relfrozenxid) AS tx_for_wraparound_vacuum
FROM pg_class
WHERE relfrozenxid != 0
AND oid > 16384
ORDER BY tx_for_wraparound_vacuum ;
</span></code></pre>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><i><br /></i>
But, lets assume that you got to the point that you started seeing<b> "autovacuum: VACUUM tablename (to prevent wraparound)"</b></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">This vacuum will more likely happen when you don't want it and even if you kill it, it will keep respawning.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">If you already set autovacuum_freeze_max_age to a more viable production setting, we usually set it at 1.5bil, you can change autovacuum_freeze_max_age to a higher value, say 3 billion and immediately kick of a manual vacuum on the table with a high maintenance_work_mem value. This vacuum will be a "normal vacuum" hence live. If the table is so big that each manual vacuum take days to complete, then you should've partitioned it...Proper schema and proper tuning of autovacuum are really important, especially in write heavy workloads.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">And to get back to the outage, the mail from the company insinuates that it was a problem with postgres, reality is that it wasn't, it was clearly an OPS oversight.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">Thanks for reading</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">Vasilis Ventirozos</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">credativ LLC</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com4tag:blogger.com,1999:blog-826910957631479390.post-47834020809602254652019-01-25T02:44:00.000-08:002019-02-04T08:44:24.815-08:00PostgreSQL 12 : Where in copy from5 days ago a commit made it to 12devel that implements WHERE clause in COPY FROM.<br />
Today we're gonna see how it works and see how someone could achieve the same by using file_fdw.<br />
<br />
To begin with, lets create a table and put some data in it.<br />
<br />
create table test (<br />
id int,<br />
date timestamp without time zone,<br />
);<br />
<br />
insert into test (id,date) select generate_series (1,1000000) ,'2015-01-01';<br />
insert into test (id,date) select generate_series (1000001,2000000),'2016-01-01';<br />
<div>
insert into test (id,date) select generate_series (2000001,3000000),'2017-01-01';</div>
<div>
insert into test (id,date) select generate_series (3000001,4000000),'2018-01-01';</div>
<div>
insert into test (id,date) select generate_series (4000001,5000000),'2019-01-01';</div>
<div>
<br /></div>
<div>
now, lets make this a bit larger than 170MB, dump the data in csv and truncate the table :</div>
<div>
<div>
<br /></div>
<div>
monkey=# insert into test select * from test;</div>
<div>
INSERT 0 5000000</div>
<div>
monkey=# insert into test select * from test;</div>
<div>
INSERT 0 10000000</div>
<div>
monkey=# select count(*) from test ;</div>
<div>
count</div>
<div>
----------</div>
<div>
20000000</div>
<div>
(1 row)</div>
</div>
<div>
<br /></div>
<div>
<div>
monkey=# copy test to '/home/vasilis/test.csv' with csv;</div>
<div>
COPY 20000000</div>
<div>
<br /></div>
<div>
monkey=# truncate test;</div>
<div>
TRUNCATE TABLE</div>
</div>
<div>
<br /></div>
<div>
<div>
vasilis@Wrath > ls -lh ~/test.csv</div>
<div>
-rw-r--r-- 1 vasilis vasilis 759M Jan 25 12:24 /home/vasilis/test.csv</div>
</div>
<div>
<br /></div>
<div>
Our test file is about 750Mb, now with an empty table , lets import only the rows that are up to 2 years old :</div>
<div>
<br /></div>
<div>
<div>
monkey=# copy test from '/home/vasilis/test.csv' with csv where date >= now() - INTERVAL '2 year' ;</div>
<div>
COPY 8000000</div>
<div>
<b>Time: 17770.267 ms (00:17.770)</b></div>
</div>
<div>
<b><br /></b></div>
<div>
It worked , awesome !</div>
<div>
<br /></div>
<div>
Now, lets compare to the alternative, file_fdw :</div>
<div>
<br /></div>
<div>
monkey=# CREATE EXTENSION file_fdw;</div>
<div>
<div>
CREATE EXTENSION</div>
<div>
Time: 7.288 ms</div>
<div>
<br /></div>
<div>
monkey=# CREATE SERVER pgtest FOREIGN DATA WRAPPER file_fdw;</div>
<div>
CREATE SERVER</div>
<div>
Time: 3.957 ms</div>
<div>
<br /></div>
<div>
monkey=# create foreign table file_test (id int, date timestamp without time zone, name text) server pgtest options (filename '/home/vasilis/test.csv', format 'csv');</div>
<div>
CREATE FOREIGN TABLE</div>
<div>
Time: 16.463 ms</div>
<div>
<br /></div>
<div>
monkey=# truncate test ;</div>
<div>
TRUNCATE TABLE</div>
<div>
Time: 15.123 ms</div>
</div>
<div>
<div>
<b><br /></b></div>
<div>
<b>monkey=# insert into test select * from file_test where date >= now() - INTERVAL '2 year' ;</b></div>
<div>
<b>INSERT 0 8000000</b></div>
<div>
<b>Time: 21368.595 ms (00:21.369)</b></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
I automated the test, and copy was (obviously) always faster. I'd imagine that the key benefit here is not performance, which is inherited by COPY itself but extending the functionality of COPY, which is good, especially for workloads that export data , ETL and load to another DB, or for restoring specific rows from backups (assuming pg_dump backups is still a thing).</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Thanks for reading</div>
<div>
Vasilis Ventirozos</div>
<div>
credativ LLC<br />
<br /></div>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com1tag:blogger.com,1999:blog-826910957631479390.post-17895221352576022162018-11-12T14:17:00.002-08:002018-11-12T14:17:47.292-08:00Zero downtime upgrade PostgreSQL 10 to 11.<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="font-family: inherit; font-size: large; font-weight: normal;">PostgreSQL 11 has been officially released and it's packed with exciting changes.Native declarative partitioning is more robust, parallelism is now allowing moreoperations to work in parallel, transactions are now supported in stored procedures and just in time compilation for expressions are just some of the new features.</span></h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="font-weight: normal;"><span style="font-family: inherit; font-size: large;">With every major version, Postgres is getting better, providing more and more tools to </span><span style="font-family: inherit; font-size: large;">developers and better quality of life to the DBAs, making it harder to resist an upgrade.</span><span style="font-family: inherit; font-size: large;">That said, while new major versions come with all these exciting improvements, they also come with internal system changes, making the database clusters incompatible across major versions. This means that upgrades up to Postgres 10, where logical replication was introduced, were impossible without downtime or 3rd party replication tools.</span></span></h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="font-weight: normal;"><span style="font-family: inherit; font-size: large;"><br /></span></span></h1>
<div>
<span style="font-size: large;"><b>Story time,</b></span></div>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="font-weight: normal;"><span style="font-family: inherit; font-size: large;">Before we get into how to upgrade using logical replication, let's see how upgrading and replication evolved over the years. </span><span style="font-family: inherit; font-size: large;">pg_dump and pg_restore is probably the most traditional way to do an upgrade. This method </span><span style="font-family: inherit; font-size: large;">requires all database writes to be suspended, making it impractical for any reasonably sized </span><span style="font-family: inherit; font-size: large;">production database. Pg_upgrade was introduced in Postgres 8.4, it’s still the most common way to do upgrades. It works under the assumption that the internal storage format rarely changes allowing it to create new system catalog tables and simply reuse the old data files. This means that upgrades are safe and fast. It still requires the database to be down and by default it will copy the datafiles to a new data directory, This can take significant amount of time but it can easily bypassed by using the hard link option provided by pg_upgrade itself. Hard links are only valid in the same filesystem, and with that in mind, this method not only massively reduces downtime, but also eliminates the need of having a second copy of the database cluster. In rare occasions like for example, an upgrade changing storage options, like floating point to int64 date/times, pg_upgrade won’t help, it will exit saying that clusters are incompatible. For this reason, or in case a database downtime -even a brief one- is inhibitory, there are third party tools that can perform upgrades with zero downtime. pg_logical and Slony are the most popular ones, with Slony doing trigger based replication and pg_logical using logical decoding.</span></span></h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="font-family: inherit; font-size: large; font-weight: normal;"> Just to give you a sense of how different the downtime requirements can be for each upgrade method below there are </span><span style="font-family: inherit; font-size: large; font-weight: normal;">timings for a 13GB database using each of the three methods we discussed above:</span></h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<ul>
<li><span style="font-family: inherit; font-size: large;">pg_dump : 10m52.363s</span></li>
<li><span style="font-family: inherit; font-size: large;">pg_upgrade : 2m41.983s</span></li>
<li><span style="font-family: inherit; font-size: large;">pg_upgrade using hard links : 0m4.545s</span></li>
</ul>
</h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="font-family: inherit; font-size: large;">Replication</span></h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="font-weight: normal;"><span style="font-family: inherit; font-size: large;">Introduced in 8.3, warm standbys were the first Postgres native replication method. Based on log shipping where WAL changes, packed into WAL files were shipped directly from one server to another in order to be replayed sequentially one file at the time. </span><span style="font-family: inherit; font-size: large;">Streaming replication was introduced at 9.0, this gave us the ability to transmit WAL records instead of WAL files over a separate and dedicated replication communication channel. It is however replicating changes byte-per-byte so it doesn’t allow any writes in standbys. Also partial replication of individual databases or objects isn’t possible. </span><span style="font-family: inherit; font-size: large;">In Postgres 10 Logical replication was introduced and while it uses the same communication channel and the same data as streaming, it uses Logical Decoding feature to make logical changes to the target database based on each objects replica identity, usually a primary key. </span><span style="font-family: inherit; font-size: large;">Logical replication uses a publisher-subscriber model where one or more subscribers subscribe to one or more publications on a publisher node. Replication of an object usually starts by transferring a snapshot of the object's data to the subscriber. Once this is done the changes are being transmitted as they happen through the native replication channel over a replication slot that is automatically created. </span><span style="font-family: inherit; font-size: large;">Because logical replication is sending incremental changes of individual objects it gives flexibility that doesn't exist in Streaming replication :</span></span></h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<ul>
<li><span style="font-family: inherit; font-size: large; font-weight: normal;">It replicates between different major versions</span></li>
<li><span style="font-family: inherit; font-size: large; font-weight: normal;">It replicates to a fully functional and writable target database</span></li>
<li><span style="font-family: inherit; font-size: large; font-weight: normal;">It works on different platforms (Linux to Windows)</span></li>
</ul>
</h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="font-family: inherit; font-size: large;"> Upgrading by replicating.</span></h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="font-size: large;"><span style="font-family: inherit;"><span style="font-weight: normal;">With cloud installations getting very popular and with AWS RDS supporting Logical replication since Jul 25, 2018 one could easily make a transition from Postgres 10 to 11 with absolutely zero database downtime. The only downtime needed would be the application switchover to the new database. The process in theory is pretty straight forward, you need to create a publisher on your postgres 10, adding all your tables to one or more publications, transfer an empty schema to postgres 11 and create a subscriber that will subscribe to the publication(s). We won’t get into the details of each individual command, there are numerous guides that explain the process. Here’s a video of </span></span><a href="https://xzilla.net/writings" style="font-family: inherit; font-weight: normal;">Robert Treat</a><span style="font-family: inherit;"><span style="font-weight: normal;">, demonstrating Logical replication live during </span></span><a href="https://www.youtube.com/watch?v=YpsJu2mtBKA" style="font-family: inherit; font-weight: normal;">Postgres Open SV 2017</a><span style="font-family: inherit;"><span style="font-weight: normal;">. Instead we are going to talk about the possibilities that this method could offer now or in the near future. Over the last 15 years or so, we moved from dump-restore offline upgrades, to pg_upgrade (rapid, offline upgrades) to logical replication upgrades, pg_upgrade served us really well, but it has its cons. You have to generate new database statistics after an upgrade and this, when sizes are big or statistics targets are tuned higher than the defaults can be time consuming. Yes, this can be done online but until it's done, database performance is compromised. Also, the upgraded database will start with cold cache and sometimes it'll take hours or even days to get to the state it was. Upgrading with logical replication gives us the ability to update statistics and pre-warm cache on the replica prior to the application switch over, something that isn't possible with pg_upgrade method. Up until 11 </span></span><span style="font-weight: normal;">emergency</span><span style="font-family: inherit;"><span style="font-weight: normal;"> downgrades were not a real option, now with logical replication they are. Lastly, as we said earlier, pg_upgrade works under the assumption that internal storage format doesn't change and even if the community will try its best not to let that happen, pluggable storage engines are a possible future feature, and if this ever happens, i'm not sure how pg_upgrade could work. All these restrictions are in theory lifted with logical replication, and assuming that tools around logical replication will be developed I can see a future where we no longer upgrade databases, we just make transitions to between versions.</span></span></span></h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="font-family: inherit; font-size: large;">Caveats - word of advice</span></h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="font-weight: normal;"><span style="font-family: inherit; font-size: large;">Logical replication offers a lot of flexibility but this doesn’t come without a <a href="https://www.postgresql.org/docs/11/logical-replication-restrictions.html">cost</a>, it’s very fragile to changes and it can be quite tricky to replicate a database with a complicated schema. It also requires every single replicated table to have a primary key and it has much more maintenance work than Streaming replication, which remains the goto solution for high availability uses. Another thing is that logical replication lifts the restriction of having a single point of writes and this, if not handled well from the application can end up causing split brain situations, where you have parts of the application writing to two different masters at the same time. This can be easily solved with DNS , virtual ips or by using connection pooling but it’s still more prone to mistakes than the other upgrade methods. </span><span style="font-family: inherit; font-size: large;">Logical replication is a relatively new postgres feature, and as most of the new features it lacks the proper tooling, with postgres 11 being the first version that can be upgraded using logical replication, this method is still in its infancy. As mentioned before, there are certain use cases where logical replication can help, but I also need to mention that pg_upgrade is extremely mature, it's safe and it should be the recommended way of upgrading your postgres cluster.</span></span></h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="caret-color: rgb(204, 204, 204); font-weight: normal;"><span style="font-family: inherit; font-size: large;"><br /></span></span></h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="caret-color: rgb(204, 204, 204); font-weight: normal;"><span style="font-family: inherit; font-size: large;">Thanks for reading.</span></span></h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<span style="font-family: inherit; font-size: large;"><span style="background-color: #333333; caret-color: rgb(204, 204, 204); font-weight: normal;"></span></span><div class="" dir="ltr" style="caret-color: rgb(204, 204, 204); font-weight: normal;">
<span style="font-family: inherit; font-size: large;">Vasilis Ventirozos</span></div>
<div class="" dir="ltr" style="caret-color: rgb(204, 204, 204); font-weight: normal;">
<span style="font-family: inherit; font-size: large;">OmniTI Computer Consulting</span></div>
</h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<br /></h1>
<h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;">
<br /></h1>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com2tag:blogger.com,1999:blog-826910957631479390.post-14921637355728228002018-09-25T06:51:00.000-07:002018-11-12T14:20:52.132-08:00Logical Replication in aws RDS<span style="font-family: inherit; font-size: large;">Recently Amazon annnounced that postgres 10 supports logical replication syntax and that it can work as either a publisher or a subscriber. The announcement can be found <a href="https://aws.amazon.com/about-aws/whats-new/2018/07/rds-postgresql-supports-minor-versions/">here</a>.</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">In order to test it out i made 2 10.4 RDS instances. Set them up to be in the same security,</span><br />
<span style="font-family: inherit; font-size: large;">subnet and parameter groups so i ended up with something like this :</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">user : vasilis</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">password : Ap4ssw0rd</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">db instance : lrnode1 & 2</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">dbname : lrtest</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">port : 5432</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;"><br class="" /></span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">only thing i had to change in order to enable logical replication is to change <b>rds.enable_logical_replication </b>to 1 in my parameter group.</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
</div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;"><br /></span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">After verifying connectivity</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;"><br /></span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">psql -h <a class="" href="http://lrnode1.cqrcyb3h2bq2.us/">lrnode1.random.us</a>-east-1.rds.amazonaws.com -U vasilis -d lrtest -W</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">psql -h <a class="" href="http://lrnode2.cqrcyb3h2bq2.us/">lrnode2.random.us</a>-east-1.rds.amazonaws.com -U vasilis -d lrtest -W</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;"><br class="" /></span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">i created a dummy table on my publisher and subscriber (lrnode1 and 2)</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;"><br class="" /></span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">create table lr_test_tbl (</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;"> id int primary key,</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;"> name text</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">);</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;"><br class="" /></span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">on lrnode1 (publisher) i created a publication</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">CREATE PUBLICATION lr_pub FOR TABLE lr_test_tbl ;</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;"><br class="" /></span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;"><br class="" /></span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">on lrnode2 <b>and as vasilis</b> i created the subscription</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">CREATE SUBSCRIPTION lr_sub CONNECTION 'host=<a class="" href="http://lrnode1.cqrcyb3h2bq2.us/">lrnode1.cqrcyb3h2bq2.us</a>-east-1.rds.amazonaws.com </span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;">dbname=lrtest user=vasilis password=Ap4ssw0rd' PUBLICATION lr_pub ;</span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<span style="font-family: inherit; font-size: large;"><br class="" /></span></div>
<div class="" style="font-family: Helvetica; font-size: 14px;">
<div class="" style="-webkit-nbsp-mode: space; line-break: after-white-space; word-wrap: break-word;">
<span style="font-family: inherit; font-size: large;">There only 3 things you need to know, you need to adjust your subnet / security groups allowing</span></div>
<div class="" style="-webkit-nbsp-mode: space; line-break: after-white-space; word-wrap: break-word;">
<span style="font-family: inherit; font-size: large;">instances to see each other, you have to change the RDS.enable_logical_replication parameter </span></div>
<div class="" style="-webkit-nbsp-mode: space; line-break: after-white-space; word-wrap: break-word;">
<span style="font-family: inherit; font-size: large;">and that the user you are supposed to use for this is the user you set when you created the instance.</span></div>
<div class="" style="-webkit-nbsp-mode: space; line-break: after-white-space; word-wrap: break-word;">
</div>
<div class="" style="-webkit-nbsp-mode: space; line-break: after-white-space; word-wrap: break-word;">
<span style="font-family: inherit; font-size: large;">That was it, pretty fast and simple.</span></div>
<div class="" style="-webkit-nbsp-mode: space; line-break: after-white-space; word-wrap: break-word;">
<span style="font-family: inherit; font-size: large;"><br /></span></div>
<div class="" style="-webkit-nbsp-mode: space; line-break: after-white-space; word-wrap: break-word;">
<span style="font-family: inherit; font-size: large;">-- </span><br />
<div class="gmail_signature">
<div class="" dir="ltr">
<span style="font-family: inherit; font-size: large;">Vasilis Ventirozos</span></div>
<div class="" dir="ltr">
<span style="font-family: inherit; font-size: large;">OmniTI Computer Consulting Inc.</span></div>
</div>
</div>
</div>
<br />Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com2tag:blogger.com,1999:blog-826910957631479390.post-91326820206257994822018-03-01T07:13:00.000-08:002018-03-01T07:50:27.312-08:00Implementing a "distributed" reporting server using some of postgres10 features.<span style="font-family: inherit; font-size: large;">Today i will try to show how strong Postgres 10 is by combining different features in order to create a "distributed" reporting server. The features that i will be using are :</span><br />
<ul>
<li><span style="font-family: inherit; font-size: large;">Logical Replication</span></li>
<li><span style="font-family: inherit; font-size: large;">Partitioning</span></li>
<li><span style="font-family: inherit; font-size: large;">Foreign Data Wrappers</span></li>
<li><span style="font-family: inherit; font-size: large;">Table Inheritance </span></li>
</ul>
<span style="font-family: inherit; font-size: large;">The scenario that we want to implement is the following :<span style="font-size: xx-small;"> </span></span><br />
<div>
<span style="font-family: inherit; font-size: large;">We have one central point for inserts, that we will call Bucket, bucket is partitioned by range yearly.In my example we have 3 partitions for 2016, 2017, 2018 and each partition is logically replicated to 3 data nodes, each responsible for 1 year of data. Finally we have a reporting proxy that is responsible for all selects and connects to each node through foreign data wrappers.</span></div>
<div>
<span style="font-family: inherit; font-size: large;">The setup consists in 5 docker containers that have the following roles.</span><br />
<ul>
<li><span style="font-family: inherit; font-size: large;">10.0.0.2, bucket, insert / update / delete</span></li>
<li><span style="font-family: inherit; font-size: large;">10.0.0.3, node2016, data holder for 2016</span></li>
<li><span style="font-family: inherit; font-size: large;">10.0.0.4, node2017, data holder for 2017</span></li>
<li><span style="font-family: inherit; font-size: large;">10.0.0.5, node2018, data holder for 2018</span></li>
<li><span style="font-family: inherit; font-size: large;">10.0.0.6, reporting proxy, main point for selects </span></li>
</ul>
<span style="font-family: inherit; font-size: large;"><span style="font-size: xx-small;"><br /></span>
</span><br />
<div>
<span style="font-family: inherit; font-size: large;">Now lets start with the bucket :</span><br />
<span style="font-family: inherit; font-size: large;"><br />
<!-- HTML generated using hilite.me --></span><br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;"><span style="color: #66d9ef;">CREATE</span> <span style="color: #66d9ef;">TABLE</span> <span style="color: #f8f8f2;">data_bucket</span> <span style="color: #f8f8f2;">(</span>
<span style="color: #f8f8f2;"> </span><span style="color: #f8f8f2;">id</span> <span style="color: #f8f8f2;">int</span> <span style="color: #f8f8f2;">,</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">data</span> <span style="color: #f8f8f2;">text,</span>
<span style="color: #f8f8f2;"> </span><span style="color: #f8f8f2;">insert_time</span> <span style="color: #66d9ef;">timestamp</span> <span style="color: #66d9ef;">without</span> <span style="color: #f8f8f2;">time</span> <span style="color: #66d9ef;">zone</span> <span style="color: #66d9ef;">DEFAULT</span> <span style="color: #f8f8f2;">now())</span>
<span style="color: #f8f8f2;"> </span><span style="color: #f8f8f2;">PARTITION</span> <span style="color: #66d9ef;">BY</span> <span style="color: #f8f8f2;">RANGE</span> <span style="color: #f8f8f2;">(insert_time);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #66d9ef;">TABLE</span> <span style="color: #f8f8f2;">data_p2016</span> <span style="color: #f8f8f2;">PARTITION</span> <span style="color: #66d9ef;">OF</span> <span style="color: #f8f8f2;">data_bucket</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">FOR</span> <span style="color: #66d9ef;">VALUES</span> <span style="color: #66d9ef;">FROM</span> <span style="color: #f8f8f2;">(</span><span style="color: #e6db74;">'2016-01-01 00:00:00'</span><span style="color: #f8f8f2;">)</span> <span style="color: #66d9ef;">TO</span> <span style="color: #f8f8f2;">(</span><span style="color: #e6db74;">'2017-01-01 00:00:00'</span><span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #66d9ef;">TABLE</span> <span style="color: #f8f8f2;">data_p2017</span> <span style="color: #f8f8f2;">PARTITION</span> <span style="color: #66d9ef;">OF</span> <span style="color: #f8f8f2;">data_bucket</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">FOR</span> <span style="color: #66d9ef;">VALUES</span> <span style="color: #66d9ef;">FROM</span> <span style="color: #f8f8f2;">(</span><span style="color: #e6db74;">'2017-01-01 00:00:00'</span><span style="color: #f8f8f2;">)</span> <span style="color: #66d9ef;">TO</span> <span style="color: #f8f8f2;">(</span><span style="color: #e6db74;">'2018-01-01 00:00:00'</span><span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #66d9ef;">TABLE</span> <span style="color: #f8f8f2;">data_p2018</span> <span style="color: #f8f8f2;">PARTITION</span> <span style="color: #66d9ef;">OF</span> <span style="color: #f8f8f2;">data_bucket</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">FOR</span> <span style="color: #66d9ef;">VALUES</span> <span style="color: #66d9ef;">FROM</span> <span style="color: #f8f8f2;">(</span><span style="color: #e6db74;">'2018-01-01 00:00:00'</span><span style="color: #f8f8f2;">)</span> <span style="color: #66d9ef;">TO</span> <span style="color: #f8f8f2;">(</span><span style="color: #e6db74;">'2019-01-01 00:00:00'</span><span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">unique</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">data_p2016_uniq</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_p2016</span> <span style="color: #f8f8f2;">(id);</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">unique</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">data_p2017_uniq</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_p2017</span> <span style="color: #f8f8f2;">(id);</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">unique</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">data_p2018_uniq</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_p2018</span> <span style="color: #f8f8f2;">(id);</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">data_p2016_time</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_p2016</span> <span style="color: #f8f8f2;">(insert_time);</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">data_p2017_time</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_p2017</span> <span style="color: #f8f8f2;">(insert_time);</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">data_p2018_time</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_p2018</span> <span style="color: #f8f8f2;">(insert_time);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #f8f8f2;">PUBLICATION</span> <span style="color: #f8f8f2;">pub_data_p2016</span> <span style="color: #66d9ef;">FOR</span> <span style="color: #66d9ef;">TABLE</span> <span style="color: #f8f8f2;">data_p2016</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">WITH</span> <span style="color: #f8f8f2;">(publish</span><span style="color: #f92672;">=</span><span style="color: #e6db74;">'insert,update'</span><span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #f8f8f2;">PUBLICATION</span> <span style="color: #f8f8f2;">pub_data_p2017</span> <span style="color: #66d9ef;">FOR</span> <span style="color: #66d9ef;">TABLE</span> <span style="color: #f8f8f2;">data_p2017</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">WITH</span> <span style="color: #f8f8f2;">(publish</span><span style="color: #f92672;">=</span><span style="color: #e6db74;">'insert,update'</span><span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #f8f8f2;">PUBLICATION</span> <span style="color: #f8f8f2;">pub_data_p2018</span> <span style="color: #66d9ef;">FOR</span> <span style="color: #66d9ef;">TABLE</span> <span style="color: #f8f8f2;">data_p2018</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">WITH</span> <span style="color: #f8f8f2;">(publish</span><span style="color: #f92672;">=</span><span style="color: #e6db74;">'insert,update'</span><span style="color: #f8f8f2;">);</span>
</span></pre>
</div>
<span style="font-family: inherit; font-size: large;">
<br />
</span></div>
<div>
<span style="font-family: inherit; font-size: large;">Here we created a data bucket table that we will insert into, its yearly partitions, some indexes for uniqueness and for searching dates, indexes are optional, more about that later on. Last we created 3 publications that we will use in our next step. Notice that we only replicate inserts and updates not deletes. Just keep that in mind for later.</span></div>
<div>
<span style="font-family: inherit; font-size: large;"><br /></span></div>
<div>
<span style="font-family: inherit; font-size: large;">Next step is setting up the data nodes. On each container (node2016, node2017 and node2018) run the following SQL :</span></div>
<div>
<!-- HTML generated using hilite.me --><span style="font-family: inherit; font-size: large;"><span style="font-size: xx-small;"><br /></span>
</span><br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;"><span style="color: #75715e;">-- node 2016</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #66d9ef;">TABLE</span> <span style="color: #f8f8f2;">data_p2016</span> <span style="color: #f8f8f2;">(</span>
<span style="color: #f8f8f2;"> id</span> <span style="color: #f8f8f2;">int,</span>
<span style="color: #66d9ef;"> data</span> <span style="color: #f8f8f2;">text,</span>
<span style="color: #f8f8f2;"> insert_time</span> <span style="color: #66d9ef;">timestamp</span> <span style="color: #66d9ef;">without</span> <span style="color: #f8f8f2;">time</span> <span style="color: #66d9ef;">zone</span> <span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">unique</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">data_p2016_uniq</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_p2016</span> <span style="color: #f8f8f2;">(id);</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">data_p2016_time</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_p2016</span> <span style="color: #f8f8f2;">(insert_time);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #f8f8f2;">SUBSCRIPTION</span> <span style="color: #f8f8f2;">sub_data_p2016</span> </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;"><span style="color: #66d9ef;"> CONNECTION</span> <span style="color: #e6db74;">'dbname=monkey host=10.0.0.2 user=postgres port=5432'</span>
<span style="color: #f8f8f2;"> PUBLICATION</span> <span style="color: #f8f8f2;">pub_data_p2016;</span>
<span style="color: #75715e;">-- node 2017</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #66d9ef;">TABLE</span> <span style="color: #f8f8f2;">data_p2017</span> <span style="color: #f8f8f2;">(</span>
<span style="color: #f8f8f2;"> id</span> <span style="color: #f8f8f2;">int,</span>
<span style="color: #66d9ef;"> data</span> <span style="color: #f8f8f2;">text,</span>
<span style="color: #f8f8f2;">insert_time</span> <span style="color: #66d9ef;">timestamp</span> <span style="color: #66d9ef;">without</span> <span style="color: #f8f8f2;">time</span> <span style="color: #66d9ef;">zone</span> <span style="color: #f8f8f2;">)</span> <span style="color: #f8f8f2;">;</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">unique</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">data_p2017_uniq</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_p2017</span> <span style="color: #f8f8f2;">(id);</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">data_p2017_time</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_p2017</span> <span style="color: #f8f8f2;">(insert_time);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #f8f8f2;">SUBSCRIPTION</span> <span style="color: #f8f8f2;">sub_data_p2017</span> </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;"><span style="color: #66d9ef;"> CONNECTION</span> <span style="color: #e6db74;">'dbname=monkey host=10.0.0.2 user=postgres port=5432'</span>
<span style="color: #f8f8f2;"> PUBLICATION</span> <span style="color: #f8f8f2;">pub_data_p2017;</span>
<span style="color: #75715e;">-- node 2018</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #66d9ef;">TABLE</span> <span style="color: #f8f8f2;">data_p2018</span> <span style="color: #f8f8f2;">(</span>
<span style="color: #f8f8f2;"> id</span> <span style="color: #f8f8f2;">int,</span>
<span style="color: #66d9ef;"> data</span> <span style="color: #f8f8f2;">text,</span>
<span style="color: #f8f8f2;"> insert_time</span> <span style="color: #66d9ef;">timestamp</span> <span style="color: #66d9ef;">without</span> <span style="color: #f8f8f2;">time</span> <span style="color: #66d9ef;">zone</span> <span style="color: #f8f8f2;">)</span> <span style="color: #f8f8f2;">;</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">unique</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">data_p2018_uniq</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_p2017</span> <span style="color: #f8f8f2;">(id);</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">data_p2018_time</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_p2017</span> <span style="color: #f8f8f2;">(insert_time);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #f8f8f2;">SUBSCRIPTION</span> <span style="color: #f8f8f2;">sub_data_p2018</span> </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;"><span style="color: #66d9ef;"> CONNECTION</span> <span style="color: #e6db74;">'dbname=monkey host=10.0.0.2 user=postgres port=5432'</span>
<span style="color: #f8f8f2;"> PUBLICATION</span> <span style="color: #f8f8f2;">pub_data_p2018;</span>
</span></pre>
</div>
<span style="font-family: inherit; font-size: large;"><span style="font-size: xx-small;"><br /></span>
Here, for each node we create the data table, indexes and a subscription pointing to the bucket server.</span><br />
<span style="font-family: inherit; font-size: large;">Right now every row that gets into the bucket is being transferred to the appropriate node. One last thing is missing, putting everything together. For aggregating all nodes we have the reporting proxy container. In this server we need to run the following SQL statements :</span><br />
<!-- HTML generated using hilite.me --><span style="font-family: inherit; font-size: large;"><br /></span>
<br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;"><span style="color: #66d9ef;">create</span> <span style="color: #f8f8f2;">extension</span> <span style="color: #f8f8f2;">if</span> <span style="color: #66d9ef;">not</span> <span style="color: #66d9ef;">exists</span> <span style="color: #f8f8f2;">postgres_fdw;</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #f8f8f2;">SERVER</span> <span style="color: #f8f8f2;">data_node_2016</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">FOREIGN</span> <span style="color: #66d9ef;">DATA</span> <span style="color: #f8f8f2;">WRAPPER</span> <span style="color: #f8f8f2;">postgres_fdw</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">OPTIONS</span><span style="color: #f8f8f2;">(</span><span style="color: #66d9ef;">host</span> <span style="color: #e6db74;">'10.0.0.3'</span><span style="color: #f8f8f2;">,port</span> <span style="color: #e6db74;">'5432'</span><span style="color: #f8f8f2;">,dbname</span> <span style="color: #e6db74;">'monkey'</span><span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #f8f8f2;">SERVER</span> <span style="color: #f8f8f2;">data_node_2017</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">FOREIGN</span> <span style="color: #66d9ef;">DATA</span> <span style="color: #f8f8f2;">WRAPPER</span> <span style="color: #f8f8f2;">postgres_fdw</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">OPTIONS</span><span style="color: #f8f8f2;">(</span><span style="color: #66d9ef;">host</span> <span style="color: #e6db74;">'10.0.0.4'</span><span style="color: #f8f8f2;">,port</span> <span style="color: #e6db74;">'5432'</span><span style="color: #f8f8f2;">,dbname</span> <span style="color: #e6db74;">'monkey'</span><span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #f8f8f2;">SERVER</span> <span style="color: #f8f8f2;">data_node_2018</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">FOREIGN</span> <span style="color: #66d9ef;">DATA</span> <span style="color: #f8f8f2;">WRAPPER</span> <span style="color: #f8f8f2;">postgres_fdw</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">OPTIONS</span><span style="color: #f8f8f2;">(</span><span style="color: #66d9ef;">host</span> <span style="color: #e6db74;">'10.0.0.5'</span><span style="color: #f8f8f2;">,port</span> <span style="color: #e6db74;">'5432'</span><span style="color: #f8f8f2;">,dbname</span> <span style="color: #e6db74;">'monkey'</span><span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #66d9ef;">USER</span> <span style="color: #f8f8f2;">MAPPING</span> <span style="color: #66d9ef;">FOR</span> <span style="color: #f8f8f2;">postgres</span> <span style="color: #f8f8f2;">SERVER</span> <span style="color: #f8f8f2;">data_node_2016</span> <span style="color: #66d9ef;">OPTIONS</span><span style="color: #f8f8f2;">(</span><span style="color: #66d9ef;">user</span> <span style="color: #e6db74;">'postgres'</span><span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #66d9ef;">USER</span> <span style="color: #f8f8f2;">MAPPING</span> <span style="color: #66d9ef;">FOR</span> <span style="color: #f8f8f2;">postgres</span> <span style="color: #f8f8f2;">SERVER</span> <span style="color: #f8f8f2;">data_node_2017</span> <span style="color: #66d9ef;">OPTIONS</span><span style="color: #f8f8f2;">(</span><span style="color: #66d9ef;">user</span> <span style="color: #e6db74;">'postgres'</span><span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #66d9ef;">USER</span> <span style="color: #f8f8f2;">MAPPING</span> <span style="color: #66d9ef;">FOR</span> <span style="color: #f8f8f2;">postgres</span> <span style="color: #f8f8f2;">SERVER</span> <span style="color: #f8f8f2;">data_node_2018</span> <span style="color: #66d9ef;">OPTIONS</span><span style="color: #f8f8f2;">(</span><span style="color: #66d9ef;">user</span> <span style="color: #e6db74;">'postgres'</span><span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #66d9ef;">TABLE</span> <span style="color: #f8f8f2;">reporting_table</span> <span style="color: #f8f8f2;">(</span>
<span style="color: #f8f8f2;"> </span><span style="color: #f8f8f2;">id</span> <span style="color: #f8f8f2;">int,</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">data</span> <span style="color: #f8f8f2;">text,</span> </span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;"><span style="color: #f8f8f2;">insert_time</span> <span style="color: #66d9ef;">timestamp</span> <span style="color: #66d9ef;">without</span> <span style="color: #f8f8f2;">time</span> <span style="color: #66d9ef;">zone</span><span style="color: #f8f8f2;">);</span></span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;">
<span style="color: #66d9ef;">CREATE</span> <span style="color: #66d9ef;">FOREIGN</span> <span style="color: #66d9ef;">TABLE</span> <span style="color: #f8f8f2;">data_node_2016</span> <span style="color: #f8f8f2;">(</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">CHECK</span> <span style="color: #f8f8f2;">(</span> <span style="color: #f8f8f2;">insert_time</span> <span style="color: #f92672;">>=</span> <span style="color: #f8f8f2;">DATE</span> <span style="color: #e6db74;">'2016-01-01'</span> <span style="color: #66d9ef;">AND</span> <span style="color: #f8f8f2;">insert_time</span> <span style="color: #f92672;"><</span> <span style="color: #f8f8f2;">DATE</span> <span style="color: #e6db74;">'2017-01-01'</span> <span style="color: #f8f8f2;">))</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">INHERITS</span> <span style="color: #f8f8f2;">(reporting_table)</span> <span style="color: #f8f8f2;">SERVER</span> <span style="color: #f8f8f2;">data_node_2016</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">options</span> <span style="color: #f8f8f2;">(</span><span style="color: #66d9ef;">table_name</span> <span style="color: #e6db74;">'data_p2016'</span><span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #66d9ef;">FOREIGN</span> <span style="color: #66d9ef;">TABLE</span> <span style="color: #f8f8f2;">data_node_2017</span> <span style="color: #f8f8f2;">(</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">CHECK</span> <span style="color: #f8f8f2;">(</span> <span style="color: #f8f8f2;">insert_time</span> <span style="color: #f92672;">>=</span> <span style="color: #f8f8f2;">DATE</span> <span style="color: #e6db74;">'2017-01-01'</span> <span style="color: #66d9ef;">AND</span> <span style="color: #f8f8f2;">insert_time</span> <span style="color: #f92672;"><</span> <span style="color: #f8f8f2;">DATE</span> <span style="color: #e6db74;">'2018-01-01'</span> <span style="color: #f8f8f2;">))</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">INHERITS</span> <span style="color: #f8f8f2;">(reporting_table)</span> <span style="color: #f8f8f2;">SERVER</span> <span style="color: #f8f8f2;">data_node_2017</span> <span style="color: #66d9ef;">options</span> <span style="color: #f8f8f2;">(</span><span style="color: #66d9ef;">table_name</span> <span style="color: #e6db74;">'data_p2017'</span><span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">CREATE</span> <span style="color: #66d9ef;">FOREIGN</span> <span style="color: #66d9ef;">TABLE</span> <span style="color: #f8f8f2;">data_node_2018</span> <span style="color: #f8f8f2;">(</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">CHECK</span> <span style="color: #f8f8f2;">(</span> <span style="color: #f8f8f2;">insert_time</span> <span style="color: #f92672;">>=</span> <span style="color: #f8f8f2;">DATE</span> <span style="color: #e6db74;">'2018-01-01'</span> <span style="color: #66d9ef;">AND</span> <span style="color: #f8f8f2;">insert_time</span> <span style="color: #f92672;"><</span> <span style="color: #f8f8f2;">DATE</span> <span style="color: #e6db74;">'2019-01-01'</span> <span style="color: #f8f8f2;">))</span>
<span style="color: #66d9ef;"> </span><span style="color: #66d9ef;">INHERITS</span> <span style="color: #f8f8f2;">(reporting_table)</span> <span style="color: #f8f8f2;">SERVER</span> <span style="color: #f8f8f2;">data_node_2018</span> <span style="color: #66d9ef;">options</span> <span style="color: #f8f8f2;">(</span><span style="color: #66d9ef;">table_name</span> <span style="color: #e6db74;">'data_p2018'</span><span style="color: #f8f8f2;">);</span>
</span></pre>
</div>
<br />
<span style="font-size: large;">We first create the Postgres foreign data wrapper extension , create remote servers and user mappings for each data node, then create the main reporting table and finally we create three foreign tables, one for each node using table inheritance.</span><br />
<span style="font-family: inherit; font-size: large;">The structure is ready, everything is now connected and we should be good for testing. But before we test this let's describe what to expect. By inserting into data_bucket data should be replicated into yearly partitions, these partitions will be replicated to their data nodes and the reporting proxy should aggregate all nodes by using foreign scans. </span><br />
<span style="font-family: inherit; font-size: large;">Let's insert some randomly generated data by inserting into the data_bucket:</span><br />
<span style="font-family: inherit; font-size: large;"><span style="font-size: xx-small;"><br /></span>
<!-- HTML generated using hilite.me --></span><br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;"><span style="color: #66d9ef;">insert</span> <span style="color: #66d9ef;">into</span> <span style="color: #f8f8f2;">data_bucket</span>
<span style="color: #66d9ef;"> select</span> <span style="color: #f8f8f2;">generate_series(</span><span style="color: #ae81ff;">1</span><span style="color: #f8f8f2;">,</span><span style="color: #ae81ff;">1000000</span><span style="color: #f8f8f2;">), </span>
<span style="color: #f8f8f2;"> md5(random()::text),</span>
<span style="color: #66d9ef;"> timestamp</span> <span style="color: #e6db74;">'2016-01-01 00:00:00'</span> <span style="color: #f92672;">+</span> <span style="color: #f8f8f2;">random()</span> <span style="color: #f92672;">*</span>
<span style="color: #f8f8f2;"> (</span><span style="color: #66d9ef;">timestamp</span> <span style="color: #e6db74;">'2019-01-01 00:00:00'</span> <span style="color: #f92672;">-</span> <span style="color: #66d9ef;">timestamp</span> <span style="color: #e6db74;">'2016-01-01 00:00:00'</span><span style="color: #f8f8f2;">);</span>
</span></pre>
</div>
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-size: large;">Data should be distributed into all three nodes. Now from reporting_table we created in the reporting proxy we should be able to see everything, notice the explain plans :</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">
<!-- HTML generated using hilite.me --></span>
<br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;"><span style="color: #f8f8f2;">monkey</span><span style="color: #f92672;">=#</span> <span style="color: #66d9ef;">select</span> <span style="color: #66d9ef;">count</span> <span style="color: #f8f8f2;">(</span><span style="color: #f92672;">*</span><span style="color: #f8f8f2;">)</span> <span style="color: #66d9ef;">from</span> <span style="color: #f8f8f2;">reporting_table</span> <span style="color: #f8f8f2;">;</span>
<span style="color: #66d9ef;">count</span>
<span style="color: #75715e;">---------</span>
<span style="color: #ae81ff;">1000000</span>
<span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">1</span> <span style="color: #66d9ef;">row</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f8f8f2;">monkey</span><span style="color: #f92672;">=#</span> <span style="color: #66d9ef;">select</span> <span style="color: #66d9ef;">min</span> <span style="color: #f8f8f2;">(insert_time),</span><span style="color: #66d9ef;">max</span><span style="color: #f8f8f2;">(insert_time)</span> <span style="color: #66d9ef;">from</span> <span style="color: #f8f8f2;">reporting_table;</span>
<span style="color: #66d9ef;">min</span> <span style="color: #f92672;">|</span> <span style="color: #66d9ef;">max</span>
<span style="color: #75715e;">----------------------------+----------------------------</span>
<span style="color: #ae81ff;">2016</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">01</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">01</span> <span style="color: #ae81ff;">00</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">03</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">17</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">062862</span> <span style="color: #f92672;">|</span> <span style="color: #ae81ff;">2018</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">12</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">31</span> <span style="color: #ae81ff;">23</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">59</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">39</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">671967</span>
<span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">1</span> <span style="color: #66d9ef;">row</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f8f8f2;">monkey</span><span style="color: #f92672;">=#</span> <span style="color: #66d9ef;">explain</span> <span style="color: #66d9ef;">analyze</span> <span style="color: #66d9ef;">select</span> <span style="color: #66d9ef;">min</span> <span style="color: #f8f8f2;">(insert_time),</span><span style="color: #66d9ef;">max</span><span style="color: #f8f8f2;">(insert_time)</span> <span style="color: #66d9ef;">from</span> <span style="color: #f8f8f2;">reporting_table;</span>
<span style="color: #f8f8f2;">QUERY</span> <span style="color: #f8f8f2;">PLAN</span>
<span style="color: #75715e;">--------------------------------------------------------------------------------------------------------------------------------------</span>
<span style="color: #66d9ef;">Aggregate</span> <span style="color: #f8f8f2;">(cost</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">598</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">80</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">598</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">81</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">width</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">16</span><span style="color: #f8f8f2;">)</span> <span style="color: #f8f8f2;">(actual</span> <span style="color: #f8f8f2;">time</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1708</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">333</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">1708</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">334</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">loops</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f92672;">-></span> <span style="color: #f8f8f2;">Append</span> <span style="color: #f8f8f2;">(cost</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">00</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">560</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">40</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">7681</span> <span style="color: #f8f8f2;">width</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">8</span><span style="color: #f8f8f2;">)</span> <span style="color: #f8f8f2;">(actual</span> <span style="color: #f8f8f2;">time</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">466</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">1653</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">186</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1000000</span> <span style="color: #f8f8f2;">loops</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f92672;">-></span> <span style="color: #f8f8f2;">Seq</span> <span style="color: #f8f8f2;">Scan</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">reporting_table</span> <span style="color: #f8f8f2;">(cost</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">00</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">00</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">width</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">8</span><span style="color: #f8f8f2;">)</span> <span style="color: #f8f8f2;">(actual</span> <span style="color: #f8f8f2;">time</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">002</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">002</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">loops</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f92672;">-></span> <span style="color: #66d9ef;">Foreign</span> <span style="color: #f8f8f2;">Scan</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_node_2016</span> <span style="color: #f8f8f2;">(cost</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">100</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">00</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">186</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">80</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">2560</span> <span style="color: #f8f8f2;">width</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">8</span><span style="color: #f8f8f2;">)</span> <span style="color: #f8f8f2;">(actual</span> <span style="color: #f8f8f2;">time</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">464</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">544</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">597</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">334088</span> <span style="color: #f8f8f2;">loops</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f92672;">-></span> <span style="color: #66d9ef;">Foreign</span> <span style="color: #f8f8f2;">Scan</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_node_2017</span> <span style="color: #f8f8f2;">(cost</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">100</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">00</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">186</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">80</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">2560</span> <span style="color: #f8f8f2;">width</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">8</span><span style="color: #f8f8f2;">)</span> <span style="color: #f8f8f2;">(actual</span> <span style="color: #f8f8f2;">time</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">334</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">533</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">149</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">332875</span> <span style="color: #f8f8f2;">loops</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f92672;">-></span> <span style="color: #66d9ef;">Foreign</span> <span style="color: #f8f8f2;">Scan</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_node_2018</span> <span style="color: #f8f8f2;">(cost</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">100</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">00</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">186</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">80</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">2560</span> <span style="color: #f8f8f2;">width</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">8</span><span style="color: #f8f8f2;">)</span> <span style="color: #f8f8f2;">(actual</span> <span style="color: #f8f8f2;">time</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">323</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">534</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">776</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">333037</span> <span style="color: #f8f8f2;">loops</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f8f8f2;">Planning</span> <span style="color: #f8f8f2;">time:</span> <span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">220</span> <span style="color: #f8f8f2;">ms</span>
<span style="color: #f8f8f2;">Execution</span> <span style="color: #f8f8f2;">time:</span> <span style="color: #ae81ff;">1709</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">252</span> <span style="color: #f8f8f2;">ms</span>
<span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">8</span> <span style="color: #66d9ef;">rows</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f8f8f2;">monkey</span><span style="color: #f92672;">=#</span> <span style="color: #66d9ef;">select</span> <span style="color: #f92672;">*</span> <span style="color: #66d9ef;">from</span> <span style="color: #f8f8f2;">reporting_table</span> <span style="color: #66d9ef;">where</span> <span style="color: #f8f8f2;">insert_time</span> <span style="color: #f92672;">=</span> <span style="color: #e6db74;">'2016-06-21 17:59:44'</span><span style="color: #f8f8f2;">;</span>
<span style="color: #f8f8f2;">id</span> <span style="color: #f92672;">|</span> <span style="color: #66d9ef;">data</span> <span style="color: #f92672;">|</span> <span style="color: #f8f8f2;">insert_time</span>
<span style="color: #75715e;">----+------+-------------</span>
<span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #66d9ef;">rows</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f8f8f2;">monkey</span><span style="color: #f92672;">=#</span> <span style="color: #66d9ef;">select</span> <span style="color: #f92672;">*</span> <span style="color: #66d9ef;">from</span> <span style="color: #f8f8f2;">reporting_table</span> <span style="color: #66d9ef;">where</span> <span style="color: #f8f8f2;">insert_time</span> <span style="color: #f92672;">=</span> <span style="color: #e6db74;">'2016-06-21 17:59:44.154904'</span><span style="color: #f8f8f2;">;</span>
<span style="color: #f8f8f2;">id</span> <span style="color: #f92672;">|</span> <span style="color: #66d9ef;">data</span> <span style="color: #f92672;">|</span> <span style="color: #f8f8f2;">insert_time</span>
<span style="color: #75715e;">-----+----------------------------------+----------------------------</span>
<span style="color: #ae81ff;">150</span> <span style="color: #f92672;">|</span> <span style="color: #ae81ff;">27</span><span style="color: #f8f8f2;">da6c5606ea26d4ca51c6b642547d44</span> <span style="color: #f92672;">|</span> <span style="color: #ae81ff;">2016</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">06</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">21</span> <span style="color: #ae81ff;">17</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">59</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">44</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">154904</span>
<span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">1</span> <span style="color: #66d9ef;">row</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f8f8f2;">monkey</span><span style="color: #f92672;">=#</span> <span style="color: #66d9ef;">explain</span> <span style="color: #66d9ef;">analyze</span> <span style="color: #66d9ef;">select</span> <span style="color: #f92672;">*</span> <span style="color: #66d9ef;">from</span> <span style="color: #f8f8f2;">reporting_table</span> <span style="color: #66d9ef;">where</span> <span style="color: #f8f8f2;">insert_time</span> <span style="color: #f92672;">=</span> <span style="color: #e6db74;">'2016-06-21 17:59:44.154904'</span><span style="color: #f8f8f2;">;</span>
<span style="color: #f8f8f2;">QUERY</span> <span style="color: #f8f8f2;">PLAN</span>
<span style="color: #75715e;">-----------------------------------------------------------------------------------------------------------------------</span>
<span style="color: #f8f8f2;">Append</span> <span style="color: #f8f8f2;">(cost</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">00</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">125</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">17</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">7</span> <span style="color: #f8f8f2;">width</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">44</span><span style="color: #f8f8f2;">)</span> <span style="color: #f8f8f2;">(actual</span> <span style="color: #f8f8f2;">time</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">383</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">384</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">loops</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f92672;">-></span> <span style="color: #f8f8f2;">Seq</span> <span style="color: #f8f8f2;">Scan</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">reporting_table</span> <span style="color: #f8f8f2;">(cost</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">00</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">00</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">width</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">44</span><span style="color: #f8f8f2;">)</span> <span style="color: #f8f8f2;">(actual</span> <span style="color: #f8f8f2;">time</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">002</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">002</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">loops</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f8f8f2;">Filter:</span> <span style="color: #f8f8f2;">(insert_time</span> <span style="color: #f92672;">=</span> <span style="color: #e6db74;">'2016-06-21 17:59:44.154904'</span><span style="color: #f8f8f2;">::</span><span style="color: #66d9ef;">timestamp</span> <span style="color: #66d9ef;">without</span> <span style="color: #f8f8f2;">time</span> <span style="color: #66d9ef;">zone</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f92672;">-></span> <span style="color: #66d9ef;">Foreign</span> <span style="color: #f8f8f2;">Scan</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">data_node_2016</span> <span style="color: #f8f8f2;">(cost</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">100</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">00</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">125</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">17</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">6</span> <span style="color: #f8f8f2;">width</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">44</span><span style="color: #f8f8f2;">)</span> <span style="color: #f8f8f2;">(actual</span> <span style="color: #f8f8f2;">time</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">381</span><span style="color: #f8f8f2;">..</span><span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">381</span> <span style="color: #66d9ef;">rows</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">loops</span><span style="color: #f92672;">=</span><span style="color: #ae81ff;">1</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f8f8f2;">Planning</span> <span style="color: #f8f8f2;">time:</span> <span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">172</span> <span style="color: #f8f8f2;">ms</span>
<span style="color: #f8f8f2;">Execution</span> <span style="color: #f8f8f2;">time:</span> <span style="color: #ae81ff;">0</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">801</span> <span style="color: #f8f8f2;">ms</span>
<span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">6</span> <span style="color: #66d9ef;">rows</span><span style="color: #f8f8f2;">)</span>
</span></pre>
</div>
<br />
<span style="font-size: large;">Some might say that ok, but we have all the data in 2 places, which is true.. but do we actually need data in the bucket? </span><span style="font-family: inherit; font-size: large;">Answer is no, we don't , we only need them in case we need to update. Remember that we set logical replication to only replicate insert and updates? This means that we can delete whatever we want from either the bucket or its partitions, so we can have any custom data retention, we can even truncate them if we want to remove data fast.</span><br />
<span style="font-family: inherit; font-size: large;">Now, is this solution perfect ? No, it's not, foreign data wrappers are obviously slower and they can't perform all operations but with each Postgres version they are getting better.</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;"><br /></span><span style="font-family: inherit; font-size: large;">Thanks for reading.</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<div class="" dir="ltr">
<span style="font-family: inherit; font-size: large;">Vasilis Ventirozos</span></div>
<div class="" dir="ltr">
<span style="font-family: inherit; font-size: large;">OmniTI Computer Consulting</span></div>
<span style="font-size: x-small;"><br /></span></div>
</div>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com0tag:blogger.com,1999:blog-826910957631479390.post-84916081259192597262018-02-27T07:23:00.002-08:002018-03-01T08:00:48.248-08:00Postgres10 in RDS, first impressions<span style="font-family: inherit; font-size: large;">As a firm believer of Postgres, and someone who runs Postgres10 in production and runs RDS in production, I've been waiting for Postgres10 on RDS to be announced ever since the release last fall. Well, today was not that day, but I was surprised to see that RDS is now sporting a "postgres10" instance you can spin up. I'm not sure if thats there on purpose, but you can be sure I jumped at the chance to get a first look at what the new Postgres 10 RDS world might look like; here is what I found..</span><br />
<span style="font-family: inherit; font-size: large;">The first thing that i wanted to test was logical replication. By default it was disabled with <i>rds.logical_replication</i> being set to 0. AWS console allowed me to change this, which also changed <i>wal_level</i> to logical so i started creating a simple table to replicate. I created a publication that included my table but thats where the party stopped. I can't create a role with replication privilege and i can't grant replication to any user :</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<div style="background-color: #272822; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 16.25px;"><span style="font-family: inherit; font-size: large;"><span style="color: #f8f8f2;">mydb</span><span style="color: #f92672;">=></span> <span style="color: #66d9ef;">SELECT</span> <span style="color: #66d9ef;">SESSION_USER</span><span style="color: #f8f8f2;">,</span> <span style="color: #66d9ef;">CURRENT_USER</span><span style="color: #f8f8f2;">;</span>
<span style="color: #66d9ef;">session_user</span> <span style="color: #f92672;">|</span> <span style="color: #66d9ef;">current_user</span>
<span style="color: #75715e;">--------------+---------------</span>
<span style="color: #f8f8f2;">testuser</span> <span style="color: #f92672;">|</span> <span style="color: #f8f8f2;">rds_superuser</span>
<span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">1</span> <span style="color: #66d9ef;">row</span><span style="color: #f8f8f2;">)</span>
<span style="color: #f8f8f2;">Time:</span> <span style="color: #ae81ff;">143</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">554</span> <span style="color: #f8f8f2;">ms</span>
<span style="color: #f8f8f2;">omniti</span><span style="color: #f92672;">=></span> <span style="color: #66d9ef;">alter</span> <span style="color: #66d9ef;">role</span> <span style="color: #f8f8f2;">testuser</span> <span style="color: #66d9ef;">with</span> <span style="color: #f8f8f2;">replication;</span>
<span style="color: #f8f8f2;">ERROR:</span> <span style="color: #f8f8f2;">must</span> <span style="color: #f8f8f2;">be</span> <span style="color: #f8f8f2;">superuser</span> <span style="color: #66d9ef;">to</span> <span style="color: #66d9ef;">alter</span> <span style="color: #f8f8f2;">replication</span> <span style="color: #f8f8f2;">users</span>
<span style="color: #f8f8f2;">Time:</span> <span style="color: #ae81ff;">163</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">823</span> <span style="color: #f8f8f2;">ms</span>
</span></pre>
</div>
<span style="font-size: large;"><span style="font-family: inherit; font-size: medium;"><br /></span></span>
<span style="font-family: inherit; font-size: large;">On top of that, create subscription requires superuser. Basically logical replication is there but i don't see how anyone could actually use it. It's well known that RDS replicas can't exist outside RDS. I was hoping that postgres 10 and logical replication would add more flexibility on replicating methods. I don't think this will change anytime soon but maybe they will add functionality in console menus that will control logical replication in their own terms using their rdsadmin user, who knows..</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">Next thing i wanted to check was parallelism. <a href="https://omniti.com/does/postgresql">Remember how I said we run Postgres 10 in production?</a> One thing we found is that there are significant bugs around parallel query, and the only safe way to work around them at this point is to disable.</span><br />
<span style="font-family: inherit; font-size: large;">I was surprised to not only see it enabled, but in fact they are only running 10.1, which does not include a bunch of fixes that we need in our prod instances (not to mention upcoming fixes in 10.3). Presumably they will fix this once it becomes officially released, hopefully on 10.3. For now, please be nice and don't crash their servers just because you can.</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">I tried a bunch of other features and it sure looked like Postgres10. The new partitioning syntax is there and it works, as well as </span><br />
<span style="font-family: inherit; font-size: large;">scram-sha-256 . Obviously this is super new and they still have work to do, but I'm really excited about the chance to get a sneak peek and looking forward to seeing this get an official release date, maybe at pgconfus later this year?</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">Thanks for reading.</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<div class="" dir="ltr">
<span style="font-family: inherit; font-size: large;">Vasilis Ventirozos</span></div>
<div class="" dir="ltr">
<span style="font-family: inherit; font-size: large;">OmniTI Computer Consulting</span></div>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com5tag:blogger.com,1999:blog-826910957631479390.post-91189849775695797172018-01-10T11:21:00.001-08:002018-03-01T07:48:53.440-08:00AWS Aurora Postgres, not a great first impression<span style="font-family: inherit; font-size: large;">Recently we had a customer facing some memory issues with an Aurora PostgreSQL. What was happening is that while loading data or creating GiST indexes they got the following error :</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<!-- HTML generated using hilite.me --><br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;"><span style="color: #ae81ff;">2018</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">01</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">04</span> <span style="color: #ae81ff;">14</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">03</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">06</span> <span style="color: #f8f8f2;">UTC:</span><span style="color: #ae81ff;">171</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">32</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">45</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">250</span><span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">35508</span><span style="color: #f8f8f2;">):root</span><span style="color: #f92672;">@</span><span style="color: #f8f8f2;">test:[</span><span style="color: #ae81ff;">20457</span><span style="color: #f8f8f2;">]:ERROR:</span> <span style="color: #66d9ef;">out</span> <span style="color: #66d9ef;">of</span> <span style="color: #f8f8f2;">memory</span>
<span style="color: #ae81ff;">2018</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">01</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">04</span> <span style="color: #ae81ff;">14</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">03</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">06</span> <span style="color: #f8f8f2;">UTC:</span><span style="color: #ae81ff;">171</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">32</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">45</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">250</span><span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">35508</span><span style="color: #f8f8f2;">):root</span><span style="color: #f92672;">@</span><span style="color: #f8f8f2;">test:[</span><span style="color: #ae81ff;">20457</span><span style="color: #f8f8f2;">]:DETAIL:</span> <span style="color: #f8f8f2;">Failed</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">request</span> <span style="color: #66d9ef;">of</span> <span style="color: #66d9ef;">size</span> <span style="color: #ae81ff;">23</span><span style="color: #f8f8f2;">.</span>
<span style="color: #ae81ff;">2018</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">01</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">04</span> <span style="color: #ae81ff;">14</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">03</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">06</span> <span style="color: #f8f8f2;">UTC:</span><span style="color: #ae81ff;">171</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">32</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">45</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">250</span><span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">35508</span><span style="color: #f8f8f2;">):root</span><span style="color: #f92672;">@</span><span style="color: #f8f8f2;">test:[</span><span style="color: #ae81ff;">20457</span><span style="color: #f8f8f2;">]:</span><span style="color: #66d9ef;">STATEMENT</span><span style="color: #f8f8f2;">:</span> <span style="color: #66d9ef;">INSERT</span> <span style="color: #66d9ef;">INTO</span> <span style="color: #f8f8f2;">test_table1</span> <span style="color: #66d9ef;">VALUES</span>
<span style="color: #f8f8f2;">(</span><span style="color: #e6db74;">'domain1.com'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'Manual'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'[198.168.1.148,198.168.1.250]'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'t'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'f'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'2016-02-16 15:58:58.016626'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">''</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">''</span><span style="color: #f8f8f2;">),</span> <span style="color: #f8f8f2;">(</span><span style="color: #e6db74;">'domain2.com'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'Manual'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'[192.168.1.1,192.168.1.174]'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'t'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'f'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'2016-02-16 15:58:58.023136'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">''</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">''</span><span style="color: #f8f8f2;">),</span>
<span style="color: #f8f8f2;">...</span>
</span></pre>
</div>
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">At the same time they had a major drop in freeable memory :</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgu4Qggh9OVlATg0QHzE2jCeJb13X1yQQe892AsjV5ZoNEi85yiovWxrrMLC5_sNXJOHJA4f8Kr0foygCj521LXbRli0c2H6lV3eApWaFKJjawiCjEAIahz2e87l7byS1isDI3MMmCA153v/s1600/ssc.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: inherit; font-size: large;"><img border="0" data-original-height="747" data-original-width="1600" height="297" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgu4Qggh9OVlATg0QHzE2jCeJb13X1yQQe892AsjV5ZoNEi85yiovWxrrMLC5_sNXJOHJA4f8Kr0foygCj521LXbRli0c2H6lV3eApWaFKJjawiCjEAIahz2e87l7byS1isDI3MMmCA153v/s640/ssc.png" width="640" /></span></a></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit; font-size: large;"><br /></span></div>
<div class="separator" style="clear: both; text-align: left;">
<span style="font-family: inherit; font-size: large;">logs showed that :</span></div>
<br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #f8f8f2;">TopMemoryContext:</span> <span style="color: #ae81ff;">1184792</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">13</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">14024</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">5</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">1170768</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">CFuncHash:</span> <span style="color: #ae81ff;">8192</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">776</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">7416</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #66d9ef;">Type</span> <span style="color: #f8f8f2;">information</span> <span style="color: #66d9ef;">cache</span><span style="color: #f8f8f2;">:</span> <span style="color: #ae81ff;">24472</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">2</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">2840</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">21632</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">TopTransactionContext:</span> <span style="color: #ae81ff;">8192</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">7384</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">808</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">MessageContext:</span> <span style="color: #ae81ff;">8511544</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">11</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">2077632</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">6433912</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #66d9ef;">Operator</span> <span style="color: #66d9ef;">class</span> <span style="color: #66d9ef;">cache</span><span style="color: #f8f8f2;">:</span> <span style="color: #ae81ff;">8192</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">776</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">7416</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">smgr</span> <span style="color: #f8f8f2;">relation</span> <span style="color: #f8f8f2;">context:</span> <span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">0</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">smgr</span> <span style="color: #f8f8f2;">relation</span> <span style="color: #66d9ef;">table</span><span style="color: #f8f8f2;">:</span> <span style="color: #ae81ff;">24576</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">2</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">13008</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">4</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">11568</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">TransactionAbortContext:</span> <span style="color: #ae81ff;">32768</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">32728</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">40</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">Portal</span> <span style="color: #f8f8f2;">hash:</span> <span style="color: #ae81ff;">8192</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">776</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">7416</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">PortalMemory:</span> <span style="color: #ae81ff;">8192</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">7880</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">312</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">PortalHeapMemory:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">840</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">184</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">ExecutorState:</span> <span style="color: #ae81ff;">24576</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">2</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">3328</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">13</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">21248</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">GiST</span> <span style="color: #f8f8f2;">scan</span> <span style="color: #f8f8f2;">context:</span> <span style="color: #ae81ff;">49245329112</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">5881</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">138720</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">3880</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">49245190392</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">ExprContext:</span> <span style="color: #ae81ff;">8192</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">8056</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">136</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">ExprContext:</span> <span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">0</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">ExprContext:</span> <span style="color: #ae81ff;">8192</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">7464</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">728</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">Relcache</span> <span style="color: #66d9ef;">by</span> <span style="color: #f8f8f2;">OID:</span> <span style="color: #ae81ff;">24576</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">2</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">12960</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">4</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">11616</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">CacheMemoryContext:</span> <span style="color: #ae81ff;">516096</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">6</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">74648</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">441448</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">test_idx1:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">8</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">1016</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">test_idx2:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">400</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">624</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_index_indrelid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">400</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">624</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_constraint_conrelid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">400</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">624</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_attrdef_adrelid_adnum_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">264</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">760</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_db_role_setting_databaseid_rol_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">264</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">760</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_user_mapping_user_server_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">312</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">712</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_user_mapping_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_type_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">400</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">624</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_type_typname_nsp_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">312</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">712</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_ts_template_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_ts_template_tmplname_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">312</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">712</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_ts_parser_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_ts_parser_prsname_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">312</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">712</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_ts_dict_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_ts_dict_dictname_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">312</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">712</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_ts_config_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_ts_config_cfgname_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">312</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">712</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_ts_config_map_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">16</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">1008</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_transform_type_lang_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">312</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">712</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_transform_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_statistic_relid_att_inh_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">16</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">1008</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_class_relname_nsp_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">264</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">760</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_range_rngtypid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">400</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">624</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_proc_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">400</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">624</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_proc_proname_args_nsp_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">16</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">1008</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_opfamily_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_opfamily_am_name_nsp_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">16</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">1008</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_operator_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_operator_oprname_l_r_n_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">16</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">1008</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_namespace_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">400</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">624</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_namespace_nspname_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">400</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">624</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_language_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_language_name_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_foreign_table_relid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_foreign_server_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_foreign_server_name_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_foreign_data_wrapper_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_foreign_data_wrapper_name_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_event_trigger_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_event_trigger_evtname_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_enum_typid_label_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">312</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">712</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_enum_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_default_acl_role_nsp_obj_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">16</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">1008</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_conversion_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_constraint_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_conversion_name_nsp_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">312</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">712</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_conversion_default_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">16</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">1008</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_collation_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_collation_name_enc_nsp_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">16</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">1008</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_opclass_am_name_nsp_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">16</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">1008</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_cast_source_target_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">312</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">712</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_attribute_relid_attnam_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">312</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">712</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_amop_fam_strat_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">16</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">1008</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_amop_opr_fam_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">16</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">1008</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_aggregate_fnoid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_trigger_tgrelid_tgname_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">312</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">712</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_rewrite_rel_rulename_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">312</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">712</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_amproc_fam_proc_index:</span> <span style="color: #ae81ff;">3072</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">2</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">1976</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">1096</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_opclass_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">400</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">624</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_index_indexrelid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">400</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">624</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_attribute_relid_attnum_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">264</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">760</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_class_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">400</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">624</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_authid_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">400</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">624</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_auth_members_member_role_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">264</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">760</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_tablespace_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_shseclabel_object_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">16</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">1008</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_replication_origin_roname_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_database_datname_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">400</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">624</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_replication_origin_roiident_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">448</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">576</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_auth_members_role_member_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">312</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">712</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_database_oid_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">400</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">624</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">pg_authid_rolname_index:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">400</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">624</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">WAL</span> <span style="color: #f8f8f2;">record</span> <span style="color: #f8f8f2;">construction:</span> <span style="color: #ae81ff;">49768</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">2</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">6584</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">43184</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">GWAL</span> <span style="color: #f8f8f2;">record</span> <span style="color: #f8f8f2;">construction:</span> <span style="color: #ae81ff;">1024</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">456</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">568</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">PrivateRefCount:</span> <span style="color: #ae81ff;">8192</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">2840</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">5352</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">Aurora</span> <span style="color: #f8f8f2;">WAL</span> <span style="color: #f8f8f2;">Context:</span> <span style="color: #ae81ff;">8192</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">8152</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">40</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">Aurora</span> <span style="color: #f8f8f2;">File</span> <span style="color: #f8f8f2;">Context:</span> <span style="color: #ae81ff;">8192</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">7512</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">680</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">MdSmgr:</span> <span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">0</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">LOCALLOCK</span> <span style="color: #f8f8f2;">hash:</span> <span style="color: #ae81ff;">8192</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">776</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">7416</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">Timezones:</span> <span style="color: #ae81ff;">104120</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">2</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">2840</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">0</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">101280</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">ErrorContext:</span> <span style="color: #ae81ff;">8192</span> <span style="color: #f8f8f2;">total</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">1</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">8152</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">4</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">40</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #f8f8f2;">Grand</span> <span style="color: #f8f8f2;">total:</span> <span style="color: #ae81ff;">49256003552</span> <span style="color: #f8f8f2;">bytes</span> <span style="color: #66d9ef;">in</span> <span style="color: #ae81ff;">6012</span> <span style="color: #f8f8f2;">blocks;</span> <span style="color: #ae81ff;">2466216</span> <span style="color: #66d9ef;">free</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">3912</span> <span style="color: #f8f8f2;">chunks);</span> <span style="color: #ae81ff;">49253537336</span> <span style="color: #f8f8f2;">used</span>
<span style="color: #ae81ff;">2018</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">01</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">04</span> <span style="color: #ae81ff;">14</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">03</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">06</span> <span style="color: #f8f8f2;">UTC:</span><span style="color: #ae81ff;">172</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">31</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">45</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">250</span><span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">35508</span><span style="color: #f8f8f2;">):root</span><span style="color: #f92672;">@</span><span style="color: #f8f8f2;">test:[</span><span style="color: #ae81ff;">20457</span><span style="color: #f8f8f2;">]:ERROR:</span> <span style="color: #66d9ef;">out</span> <span style="color: #66d9ef;">of</span> <span style="color: #f8f8f2;">memory</span>
<span style="color: #ae81ff;">2018</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">01</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">04</span> <span style="color: #ae81ff;">14</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">03</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">06</span> <span style="color: #f8f8f2;">UTC:</span><span style="color: #ae81ff;">172</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">31</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">45</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">250</span><span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">35508</span><span style="color: #f8f8f2;">):root</span><span style="color: #f92672;">@</span><span style="color: #f8f8f2;">test:[</span><span style="color: #ae81ff;">20457</span><span style="color: #f8f8f2;">]:DETAIL:</span> <span style="color: #f8f8f2;">Failed</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">request</span> <span style="color: #66d9ef;">of</span> <span style="color: #66d9ef;">size</span> <span style="color: #ae81ff;">23</span><span style="color: #f8f8f2;">.</span>
<span style="color: #ae81ff;">2018</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">01</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">04</span> <span style="color: #ae81ff;">14</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">03</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">06</span> <span style="color: #f8f8f2;">UTC:</span><span style="color: #ae81ff;">172</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">31</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">45</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">250</span><span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">35508</span><span style="color: #f8f8f2;">):root</span><span style="color: #f92672;">@</span><span style="color: #f8f8f2;">test:[</span><span style="color: #ae81ff;">20457</span><span style="color: #f8f8f2;">]:</span><span style="color: #66d9ef;">STATEMENT</span><span style="color: #f8f8f2;">:</span> <span style="color: #66d9ef;">INSERT</span> <span style="color: #66d9ef;">INTO</span> <span style="color: #f8f8f2;">test_table1</span> <span style="color: #66d9ef;">VALUES</span>
<span style="color: #f8f8f2;">(</span><span style="color: #e6db74;">'domain1.com'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'Manual'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'[198.168.1.148,198.168.1.250]'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'t'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'f'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'2016-02-16 15:58:58.016626'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">''</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">''</span><span style="color: #f8f8f2;">),</span>
<span style="color: #f8f8f2;">(</span><span style="color: #e6db74;">'domain2.com'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'Manual'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'[192.168.1.1,192.168.1.174]'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'t'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'f'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'2016-02-16 15:58:58.023136'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">''</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">''</span><span style="color: #f8f8f2;">),</span>
<span style="color: #f8f8f2;">...</span>
</pre>
</div>
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">My first assumption was that this had something to do with GiST or with ip4r extension / datatypes combined with a GiST index.</span><br />
<span style="font-family: inherit; font-size: large;">Unfortunately all this investigation happened over a slack chat so i couldn't collect all the data i would like, the table was empty and this was affecting production so i suggested a vacuum full verbose that fixed the issue.</span><br />
<span style="font-family: inherit; font-size: large;">It has been bothering me and my colleagues that we never got to find the root cause so I started an aurora instance in AWS and tried to reproduce the problem.</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">I wrote a shell script that generated a lot of random rows and i've loaded about 4 mil rows in a table that looked like this :</span><br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #66d9ef;">CREATE</span> <span style="color: #66d9ef;">TYPE</span> <span style="color: #f8f8f2;">inet_range</span> <span style="color: #66d9ef;">AS</span> <span style="color: #f8f8f2;">RANGE</span> <span style="color: #f8f8f2;">(</span>
<span style="color: #f8f8f2;">subtype</span> <span style="color: #f92672;">=</span> <span style="color: #f8f8f2;">inet</span>
<span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">table</span> <span style="color: #f8f8f2;">gist_test</span> <span style="color: #f8f8f2;">(</span>
<span style="color: #f8f8f2;">testtext1</span> <span style="color: #f8f8f2;">citext</span> <span style="color: #66d9ef;">not</span> <span style="color: #66d9ef;">null</span><span style="color: #f8f8f2;">,</span>
<span style="color: #f8f8f2;">testtext2</span> <span style="color: #f8f8f2;">citext</span> <span style="color: #66d9ef;">not</span> <span style="color: #66d9ef;">null</span><span style="color: #f8f8f2;">,</span>
<span style="color: #f8f8f2;">ip_range</span> <span style="color: #f8f8f2;">inet_range</span> <span style="color: #66d9ef;">not</span> <span style="color: #66d9ef;">null</span><span style="color: #f8f8f2;">,</span>
<span style="color: #f8f8f2;">testbool1</span> <span style="color: #f8f8f2;">boolean</span> <span style="color: #66d9ef;">not</span> <span style="color: #66d9ef;">null</span> <span style="color: #66d9ef;">default</span> <span style="color: #66d9ef;">false</span><span style="color: #f8f8f2;">,</span>
<span style="color: #f8f8f2;">testbool2</span> <span style="color: #f8f8f2;">boolean</span> <span style="color: #66d9ef;">not</span> <span style="color: #66d9ef;">null</span> <span style="color: #66d9ef;">default</span> <span style="color: #66d9ef;">false</span><span style="color: #f8f8f2;">,</span>
<span style="color: #f8f8f2;">created_at</span> <span style="color: #66d9ef;">timestamp</span> <span style="color: #66d9ef;">without</span> <span style="color: #f8f8f2;">time</span> <span style="color: #66d9ef;">zone</span> <span style="color: #66d9ef;">not</span> <span style="color: #66d9ef;">null</span> <span style="color: #66d9ef;">default</span> <span style="color: #f8f8f2;">now(),</span>
<span style="color: #f8f8f2;">testemail1</span> <span style="color: #f8f8f2;">citext,</span>
<span style="color: #f8f8f2;">testemail2</span> <span style="color: #f8f8f2;">citext);</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">idx_test1</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">gist_test</span> <span style="color: #f8f8f2;">(testtext1);</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">idx_test2</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">gist_test</span> <span style="color: #66d9ef;">using</span> <span style="color: #f8f8f2;">GIST(ip_range);</span>
</pre>
</div>
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">I Did a bunch of tests, like bloating the gist index up to 80% and trying to insert or to use COPY to import data from external files, everything worked fine. Last i tried importing some data (about 300k rows) from my local machine with plain insert :</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<!-- HTML generated using hilite.me --><br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #66d9ef;">insert</span> <span style="color: #66d9ef;">into</span> <span style="color: #f8f8f2;">gist_test</span> <span style="color: #f8f8f2;">(testtext1,testtext2,ip_range,testbool1,testbool2,testemail1,testemail2)</span> <span style="color: #66d9ef;">values</span>
<span style="color: #f8f8f2;">(</span><span style="color: #e6db74;">'HyBrlWugbsGH4fusJ9czVneB+58IZaoG8Vx3NV0jgyQ'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'0SaIKhTlZyD2zlUFpSK0b2Y9gep6hzxO0q2crgm3iRM'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'[184.224.199.243,184.224.199.255]'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'t'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'f'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">''</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">''</span><span style="color: #f8f8f2;">),</span>
<span style="color: #f8f8f2;">...</span>
<span style="color: #f8f8f2;">...</span>
<span style="color: #f8f8f2;">(</span><span style="color: #e6db74;">'AZgoJrP+QUvhe3TkmaxqoX8ZKv3BQ4a2muvPGuJIqq8'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'rIWnCq4q7qwLWlPmbYqnk3voOQdA0cPB4AerbeV4lpA'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'[6.116.212.51,6.116.212.255]'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'t'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">'f'</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">''</span><span style="color: #f8f8f2;">,</span><span style="color: #e6db74;">''</span><span style="color: #f8f8f2;">);</span>
</pre>
</div>
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">Aurora went mental, i couldn't cancel the client getting :</span><br />
<span style="font-family: inherit; font-size: large;">Could not send cancel request: </span><br />
<span style="font-family: inherit; font-size: large;"><i>PQcancel() -- connect() failed: Connection refused</i></span><br />
<span style="font-family: inherit; font-size: large;">and after a while :</span><br />
<span style="font-family: inherit; font-size: large;"><i>psql: server closed the connection unexpectedly</i></span><br />
<span style="font-family: inherit; font-size: large;"><i><span class="Apple-tab-span" style="white-space: pre;"> </span>This probably means the server terminated abnormally</i></span><br />
<span style="font-family: inherit; font-size: large;"><i><span class="Apple-tab-span" style="white-space: pre;"> </span>before or while processing the request.</i></span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">Service was obviously dead.</span><br />
<span style="font-family: inherit; font-size: large;">The logs said :</span><br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #ae81ff;">2018</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">01</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">10</span> <span style="color: #ae81ff;">14</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">53</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">09</span> <span style="color: #f8f8f2;">UTC::</span><span style="color: #f92672;">@</span><span style="color: #f8f8f2;">:[</span><span style="color: #ae81ff;">6393</span><span style="color: #f8f8f2;">]:FATAL:</span> <span style="color: #66d9ef;">Storage</span> <span style="color: #f8f8f2;">initialization</span> <span style="color: #f8f8f2;">failed.</span> <span style="color: #66d9ef;">result</span> <span style="color: #f92672;">=</span> <span style="color: #ae81ff;">1436</span><span style="color: #f8f8f2;">,</span> <span style="color: #f8f8f2;">errno</span> <span style="color: #f92672;">=</span> <span style="color: #ae81ff;">0</span>
<span style="color: #ae81ff;">2018</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">01</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">10</span> <span style="color: #ae81ff;">14</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">53</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">09</span> <span style="color: #f8f8f2;">UTC::</span><span style="color: #f92672;">@</span><span style="color: #f8f8f2;">:[</span><span style="color: #ae81ff;">6391</span><span style="color: #f8f8f2;">]:LOG:</span> <span style="color: #f8f8f2;">Aurora</span> <span style="color: #f8f8f2;">Runtime</span> <span style="color: #f8f8f2;">process</span> <span style="color: #f8f8f2;">(PID</span> <span style="color: #ae81ff;">6393</span><span style="color: #f8f8f2;">)</span> <span style="color: #f8f8f2;">exited</span> <span style="color: #66d9ef;">with</span> <span style="color: #f8f8f2;">exit</span> <span style="color: #f8f8f2;">code</span> <span style="color: #ae81ff;">1</span>
<span style="color: #ae81ff;">2018</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">01</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">10</span> <span style="color: #ae81ff;">14</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">53</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">09</span> <span style="color: #f8f8f2;">UTC::</span><span style="color: #f92672;">@</span><span style="color: #f8f8f2;">:[</span><span style="color: #ae81ff;">6391</span><span style="color: #f8f8f2;">]:LOG:</span> <span style="color: #f8f8f2;">terminating</span> <span style="color: #66d9ef;">any</span> <span style="color: #f8f8f2;">other</span> <span style="color: #f8f8f2;">active</span> <span style="color: #f8f8f2;">server</span> <span style="color: #f8f8f2;">processes</span>
<span style="color: #ae81ff;">2018</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">01</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">10</span> <span style="color: #ae81ff;">14</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">53</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">09</span> <span style="color: #f8f8f2;">UTC::</span><span style="color: #f92672;">@</span><span style="color: #f8f8f2;">:[</span><span style="color: #ae81ff;">6391</span><span style="color: #f8f8f2;">]:FATAL:</span> <span style="color: #f8f8f2;">Can</span><span style="background-color: #1e0010; color: #960050;">'</span><span style="color: #f8f8f2;">t</span> <span style="color: #f8f8f2;">handle</span> <span style="color: #66d9ef;">storage</span> <span style="color: #f8f8f2;">runtime</span> <span style="color: #f8f8f2;">process</span> <span style="color: #f8f8f2;">crash</span>
<span style="color: #ae81ff;">2018</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">01</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">10</span> <span style="color: #ae81ff;">14</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">53</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">09</span> <span style="color: #f8f8f2;">UTC::</span><span style="color: #f92672;">@</span><span style="color: #f8f8f2;">:[</span><span style="color: #ae81ff;">6391</span><span style="color: #f8f8f2;">]:LOG:</span> <span style="color: #66d9ef;">database</span> <span style="color: #66d9ef;">system</span> <span style="color: #66d9ef;">is</span> <span style="color: #f8f8f2;">shut</span> <span style="color: #f8f8f2;">down</span>
<span style="color: #ae81ff;">2018</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">01</span><span style="color: #f92672;">-</span><span style="color: #ae81ff;">10</span> <span style="color: #ae81ff;">14</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">53</span><span style="color: #f8f8f2;">:</span><span style="color: #ae81ff;">09</span> <span style="color: #f8f8f2;">UTC::</span><span style="color: #f92672;">@</span><span style="color: #f8f8f2;">:[</span><span style="color: #ae81ff;">6751</span><span style="color: #f8f8f2;">]:LOG:</span> <span style="color: #f8f8f2;">Waiting</span> <span style="color: #66d9ef;">for</span> <span style="color: #f8f8f2;">runtime</span> <span style="color: #f8f8f2;">initialization</span> <span style="color: #f8f8f2;">complete...</span>
</pre>
</div>
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">I tried this test 3 times, two out of three the database restarted with the above message and one time i had to manually reboot the instance because it went completely unresponsive.</span><br />
<span style="font-family: inherit; font-size: large;">Now, at my home i just have 1mbit of upload so couldn't possibly send data fast enough to impact the server, cloudwatch was showing that everything was nice and low, except memory :</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: left;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxVlwrvN1AKVbzg0xVcxbJzwyHdjAy0FJb4vbXAEo4uq43dXkjLDqOSh48JKoNw0Ft2NrTje_xQwLAgJynt2ATzyBUWai27w80Nz49ss_IKGo7BhKUcFq9VtRj5QrGAPoB9ugIKtQBQDLD/s1600/Screen+Shot+2018-01-10+at+17.03.09.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><span style="font-family: inherit; font-size: large;"><img border="0" data-original-height="773" data-original-width="1600" height="308" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxVlwrvN1AKVbzg0xVcxbJzwyHdjAy0FJb4vbXAEo4uq43dXkjLDqOSh48JKoNw0Ft2NrTje_xQwLAgJynt2ATzyBUWai27w80Nz49ss_IKGo7BhKUcFq9VtRj5QrGAPoB9ugIKtQBQDLD/s640/Screen+Shot+2018-01-10+at+17.03.09.png" width="640" /></span></a></div>
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">So basically, this is my progress so far. I might try to re-test this on a normal RDS instance or to try without the indexes, or i will simply file a case to Amazon :)</span><br />
<span style="font-family: inherit; font-size: large;">Obviously this post is not meant to say that Aurora is bad, but if i was about to migrate to Aurora and take advantage of its features i'd make sure to double and triple check that the application is working as it should. This product is very new, i think it was released at Nov 2017 and some problems should be expected, especially if your schema is a bit out of the ordinary.</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;"><span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">Thanks for reading,</span></span><br />
<div class="" dir="ltr">
<span style="font-family: inherit; font-size: large;">Vasilis Ventirozos</span></div>
<div class="" dir="ltr">
<span style="font-family: inherit; font-size: large;">OmniTI Computer Consulting</span></div>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com10tag:blogger.com,1999:blog-826910957631479390.post-8512030737571975482017-12-08T07:39:00.001-08:002018-03-01T08:03:36.888-08:00Psql directly from Atom text editor.<span style="font-family: inherit; font-size: large;">Im not a big fun of postgres clients like pgadmin, not sure why, but i always favoured psql over everything.</span><br />
<span style="font-family: inherit; font-size: large;">A friend of mine wrote an atom package that brings psql into atom. Which is awesome. It's tested in linux and (briefly) in osx. In osx i only had to change the path for psql because for some reason it didn't get it from the environment. Feel free to check it out and spam him with any issues you might have.</span><br />
<span style="font-size: large;">Link for the package can be found </span><a href="https://atom.io/packages/atom-psql"><span style="font-size: large;">here</span></a><span style="font-size: large;">.</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<br />
<span style="font-size: large;">Thanks for reading.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">Vasilis Ventirozos</span><br />
<div class="" dir="ltr">
<span style="font-family: inherit; font-size: large;">OmniTI Computer Consulting</span></div>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com3tag:blogger.com,1999:blog-826910957631479390.post-64647742050526312152017-11-27T10:47:00.000-08:002018-03-01T08:18:19.559-08:00Monitoring statements in pg10<span style="font-family: inherit; font-size: large;">Recently we migrated our first client to postgres 10.1, the database used to run on 9.2.</span><br />
<span style="font-family: inherit; font-size: large;">Everything went smooth with a total downtime of 12 seconds but obviously monitoring broke so i had to fix that and i thought to share the new statements to save people some trouble.</span><br />
<span style="font-family: inherit; font-size: large;">The statements that broke had to do with the change of xlog -> wal (and location -> lsn) which mostly affected replication monitoring statements. Bellow is what i used to have on PG 9 and what i got for PG 10.</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-size: large;"><b>Replication Slot monitoring Postgres 9:</b></span><br />
<span style="font-family: inherit; font-size: large;">
<!-- HTML generated using hilite.me --></span>
<br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;"><span style="color: #66d9ef;">SELECT</span> <span style="color: #f8f8f2;">slot_name,</span> <span style="color: #66d9ef;">database</span><span style="color: #f8f8f2;">,</span> <span style="color: #f8f8f2;">active,pg_xlog_location_diff(pg_current_xlog_insert_location(),</span> <span style="color: #f8f8f2;">restart_lsn</span></span></pre>
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;"><span style="color: #66d9ef;">AS</span> <span style="color: #f8f8f2;">retained_bytes</span>
<span style="color: #66d9ef;">FROM</span> <span style="color: #f8f8f2;">pg_replication_slots;</span>
</span></pre>
</div>
<span style="font-family: inherit; font-size: large;"><b><span style="font-family: inherit; font-size: small;"><br /></span></b>
</span><br />
<b><span style="font-family: inherit; font-size: large;">Replication Slot monitoring Postgres 10:</span></b><br />
<span style="font-family: inherit; font-size: large;">
<!-- HTML generated using hilite.me --></span>
<br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;"><span style="color: #66d9ef;">SELECT</span> <span style="color: #f8f8f2;">slot_name,</span> <span style="color: #66d9ef;">database</span><span style="color: #f8f8f2;">,</span> <span style="color: #f8f8f2;">active,</span> <span style="color: #f8f8f2;">pg_wal_lsn_diff</span> <span style="color: #f8f8f2;">(pg_current_wal_insert_lsn(),</span> <span style="color: #f8f8f2;">restart_lsn)</span>
<span style="color: #66d9ef;">AS</span> <span style="color: #f8f8f2;">retained_bytes</span>
<span style="color: #66d9ef;">FROM</span> <span style="color: #f8f8f2;">pg_replication_slots;</span>
</span></pre>
</div>
<span style="font-family: inherit; font-size: large;"><b><u><span style="font-family: inherit; font-size: small;"><br /></span></u></b></span>
<b><span style="font-family: inherit; font-size: large;">Replication Lag monitoring Postgres 9 :</span></b><br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;"><span style="color: #66d9ef;">SELECT</span> <span style="color: #f8f8f2;">pg_xlog_location_diff(pg_current_xlog_insert_location(),</span> <span style="color: #f8f8f2;">flush_location)</span> <span style="color: #66d9ef;">AS</span> <span style="color: #f8f8f2;">lag_bytes,</span> <span style="color: #f8f8f2;">application_name</span>
<span style="color: #66d9ef;">FROM</span> <span style="color: #f8f8f2;">pg_stat_replication;</span>
</span></pre>
</div>
<br />
<b><span style="font-family: inherit; font-size: large;">Replication Lag monitoring Postgres 10 :</span></b>
<!-- HTML generated using hilite.me --><br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="font-family: inherit; font-size: large;"><span style="color: #66d9ef;">SELECT</span> <span style="color: #f8f8f2;">pg_wal_lsn_diff</span> <span style="color: #f8f8f2;">(pg_current_wal_flush_lsn(),</span> <span style="color: #f8f8f2;">flush_lsn</span> <span style="color: #f8f8f2;">)</span> <span style="color: #66d9ef;">AS</span> <span style="color: #f8f8f2;">lag_bytes,</span> <span style="color: #f8f8f2;">application_name</span>
<span style="color: #66d9ef;">FROM</span> <span style="color: #f8f8f2;">pg_stat_replication;</span>
</span></pre>
</div>
<span style="font-family: inherit; font-size: large;"><span style="font-family: inherit; font-size: small;"><br /></span></span>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Thanks for reading.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Vasilis Ventirozos</span><br />
<span style="font-size: large;">OmniTI Computer Consulting</span>Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com3tag:blogger.com,1999:blog-826910957631479390.post-69749889333033903012017-08-30T09:48:00.002-07:002018-03-01T08:29:38.158-08:00PostgreSQL on ZFS <span style="font-family: inherit; font-size: large;"><b style="font-family: arial, helvetica, sans-serif;">What is ZFS. </b><span style="font-family: "arial" , "helvetica" , sans-serif;">(a paste from wikipedia)</span></span><br />
<span style="font-family: inherit; font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span><span style="font-family: inherit; font-size: large;">ZFS is a combined file system and logical volume manager designed by Sun Microsystems. The features of ZFS include protection against data corruption, support for high storage capacities, efficient data compression, integration of the concepts of filesystem and volume management, snapshots and copy-on-write clones, continuous integrity checking and automatic repair, RAID-Z and native NFSv4 ACLs.</span><br />
<span style="font-family: inherit; font-size: large;"><span style="background-color: white;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></span></span>
<span style="font-family: inherit; font-size: large;">Advantages of ZFS.</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">Data integrity guaranteed.</span><br />
<span style="font-family: inherit; font-size: large;">ZFS has volume management on filesystem level, this enables copy on write. When a data block changed, it will change its location on the disk before the new write is complete. If you have a crash, that data would be damaged. ZFS does not change the location of the data until the write is verified, which means that your data is safe during a crash. ZFS also use checksums to ensure that all data is correct, every write is tested. ZFS doesn't just protect data with copy on write, it also offers additional protection by having an additional RAID level. RAID-Z3 which allows a maximum of three disk failures in a ZFS pool. Traditional RAID levels only allow two disk failures per volume. ZFS offers RAID mirrors who are usually composed of a single disk keeping its own copy. With a multi-disk mirror you can have multiple copies, which adds levels of data integrity not found in traditional RAID setups and it's also great for reads.</span><br />
<span style="font-family: inherit; font-size: large;">Performance</span><br />
<span style="font-family: inherit; font-size: large;">ZFS also allows to send writes to individual disks, instead of just the whole RAID volume. Because of this, ZFS can spread writes which is speeding up write performance. It also has algorithms that ensure that the most frequently used data blocks are kept in the fastest media, meaning that you can have just a few SSD drives as cache.</span><br />
<div style="border: 0px; margin-bottom: 15px; margin-top: 1px; outline: 0px; padding: 2px; vertical-align: baseline;">
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">Compression</span><br />
<span style="font-family: inherit; font-size: large;">Compression can be enabled on each dataset , its a transparent operation where zfs will compress and uncompress data on the fly. Because cpu operations are usually cheaper than IO and considering that compression algorithms are much smarter than they used to be the cpu overhead is usually not even noticeable, combined with the gain of compression this is an extremely useful feature for databases. The supported compression algorithms are: LZJB, LZ4, ZLE, and Gz (1-9). Today we are going to play with LZ4.</span><br />
<span style="font-family: inherit; font-size: large;">Snapshots</span><br />
<span style="font-family: inherit; font-size: large;">A snapshot is a read-only copy of a file system or volume. Snapshots can be created almost instantly, and they initially consume no additional disk space within the pool. However, as data within the active dataset changes, the snapshot consumes disk space by continuing to reference the old data, thus preventing the disk space from being freed.</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">And after this rather long intro on ZFS (that may or may not have pasted descriptions from oracle) its time to put it to work for a PostgreSQL database. In my</span><span style="font-family: inherit; font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif;"> <a href="https://evol-monkey.blogspot.com/2017/06/postgresql-on-zfs-with-bpf-tracing-on.html">previous post</a> I went through how to install postgres on a openZFS volume on Ubuntu so in this post i will be using a vm that already has a ZFS volume. </span></span></div>
<div style="border: 0px; margin-bottom: 15px; margin-top: 1px; outline: 0px; padding: 2px; vertical-align: baseline;">
<span style="font-family: inherit; font-size: large;">Enabling compression (LZ4).</span></div>
<!-- HTML generated using hilite.me --><br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #f92672;">></span> <span style="color: #f8f8f2;">zfs</span> <span style="color: #66d9ef;">set</span> <span style="color: #f8f8f2;">compression</span><span style="color: #f92672;">=</span><span style="color: #f8f8f2;">lz4</span> <span style="color: #f8f8f2;">pg_zfs</span>
</pre>
</div>
<div style="border: 0px; margin-bottom: 15px; margin-top: 1px; outline: 0px; padding: 2px; vertical-align: baseline;">
<br />
<span style="font-family: inherit; font-size: large;">Where pg_zfs is the name of the volume.</span><br />
<span style="font-family: inherit; font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">And yeah, that's pretty much it.</span></span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">Now i will initdb a new cluster create a table and insert some data :</span><br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">table</span> <span style="color: #f8f8f2;">test_compression</span> <span style="color: #f8f8f2;">(id</span> <span style="color: #f8f8f2;">int</span> <span style="color: #f8f8f2;">,</span> <span style="color: #f8f8f2;">name</span> <span style="color: #f8f8f2;">text</span> <span style="color: #66d9ef;">default</span> <span style="color: #e6db74;">'Evol'</span> <span style="color: #f8f8f2;">,</span> <span style="color: #f8f8f2;">surname</span> <span style="color: #f8f8f2;">text</span> <span style="color: #66d9ef;">default</span> <span style="color: #e6db74;">'Monkey'</span><span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">insert</span> <span style="color: #66d9ef;">into</span> <span style="color: #f8f8f2;">test_compression</span> <span style="color: #f8f8f2;">(id)</span> <span style="color: #66d9ef;">select</span> <span style="color: #f8f8f2;">generate_series</span> <span style="color: #f8f8f2;">(</span><span style="color: #ae81ff;">1</span><span style="color: #f8f8f2;">,</span><span style="color: #ae81ff;">5000000</span><span style="color: #f8f8f2;">);</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">unique</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">id_unique_idx</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">test_compression</span> <span style="color: #f8f8f2;">(id);</span>
<span style="color: #66d9ef;">create</span> <span style="color: #66d9ef;">index</span> <span style="color: #f8f8f2;">name_idx</span> <span style="color: #66d9ef;">on</span> <span style="color: #f8f8f2;">test_compression</span> <span style="color: #f8f8f2;">(name);</span>
<span style="color: #66d9ef;">checkpoint</span> <span style="color: #f8f8f2;">;</span>
<span style="color: #66d9ef;">select</span> <span style="color: #f8f8f2;">pg_size_pretty(pg_total_relation_size(</span><span style="color: #e6db74;">'test_compression'</span><span style="color: #f8f8f2;">));</span>
<span style="color: #f8f8f2;">pg_size_pretty</span>
<span style="color: #75715e;">----------------</span>
<span style="color: #ae81ff;">426</span> <span style="color: #f8f8f2;">MB</span>
</pre>
</div>
<span style="font-family: inherit; font-size: large;">now as root :</span><br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #f92672;">#</span> <span style="color: #f8f8f2;">zfs</span> <span style="color: #66d9ef;">get</span> <span style="color: #f8f8f2;">compressratio</span> <span style="color: #f8f8f2;">pg_zfs</span>
<span style="color: #f8f8f2;">NAME</span> <span style="color: #f8f8f2;">PROPERTY</span> <span style="color: #f8f8f2;">VALUE</span> <span style="color: #66d9ef;">SOURCE</span>
<span style="color: #f8f8f2;">pg_zfs</span> <span style="color: #f8f8f2;">compressratio</span> <span style="color: #ae81ff;">3</span><span style="color: #f8f8f2;">.</span><span style="color: #ae81ff;">38</span><span style="color: #f8f8f2;">x</span> <span style="color: #f92672;">-</span>
</pre>
</div>
<span style="font-family: inherit; font-size: large;"><br /></span></div>
<div style="border: 0px; margin-bottom: 15px; margin-top: 1px; outline: 0px; padding: 2px; vertical-align: baseline;">
<span style="font-family: inherit; font-size: large;">Which is pretty neat!</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">Using snapshots.</span><br />
<span style="font-family: inherit; font-size: large;">To get a snapshot of your database directory something like this would work. as root :</span><br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #f8f8f2;">psql</span> <span style="color: #f92672;">-</span><span style="color: #66d9ef;">c</span> <span style="color: #e6db74;">"select pg_start_backup('zfs_snapshot');"</span> <span style="color: #f92672;">-</span><span style="color: #f8f8f2;">U</span> <span style="color: #f8f8f2;">vasilis</span> <span style="color: #f8f8f2;">postgres</span> <span style="color: #f92672;">&&</span> <span style="color: #f8f8f2;">zfs</span> <span style="color: #f8f8f2;">snapshot</span> <span style="color: #f8f8f2;">pg_zfs</span><span style="color: #f92672;">@</span><span style="color: #f8f8f2;">blogpost</span> <span style="color: #f92672;">&&</span> <span style="color: #f8f8f2;">psql</span> <span style="color: #f92672;">-</span><span style="color: #66d9ef;">c</span> <span style="color: #e6db74;">"select pg_stop_backup();"</span> <span style="color: #f92672;">-</span><span style="color: #f8f8f2;">U</span> <span style="color: #f8f8f2;">vasilis</span> <span style="color: #f8f8f2;">postgres</span>
<span style="color: #f8f8f2;">zfs</span> <span style="color: #f8f8f2;">list</span> <span style="color: #f92672;">-</span><span style="color: #f8f8f2;">t</span> <span style="color: #f8f8f2;">snapshot</span>
<span style="color: #f8f8f2;">NAME</span> <span style="color: #f8f8f2;">USED</span> <span style="color: #f8f8f2;">AVAIL</span> <span style="color: #f8f8f2;">REFER</span> <span style="color: #f8f8f2;">MOUNTPOINT</span>
<span style="color: #f8f8f2;">pg_zfs</span><span style="color: #f92672;">@</span><span style="color: #f8f8f2;">blogpost</span> <span style="color: #ae81ff;">0</span> <span style="color: #f92672;">-</span> <span style="color: #ae81ff;">279</span><span style="color: #f8f8f2;">M</span> <span style="color: #f92672;">-</span>
</pre>
</div>
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">To restore, say that we delete the whole data directory :</span><br />
<!-- HTML generated using hilite.me --><br />
<div style="background: #272822; border-width: 0.1em 0.1em 0.1em 0.8em; border: solid gray; overflow: auto; padding: 0.2em 0.6em; width: auto;">
<pre style="line-height: 125%; margin: 0;"><span style="color: #f8f8f2;">rm</span> <span style="color: #f92672;">-</span><span style="color: #f8f8f2;">rf</span> <span style="color: #f92672;">/</span><span style="color: #f8f8f2;">pg_zfs</span><span style="color: #f92672;">/</span><span style="color: #f8f8f2;">pgdata</span><span style="color: #f92672;">/</span>
<span style="color: #f8f8f2;">killall</span> <span style="color: #f92672;">-</span><span style="color: #ae81ff;">9</span> <span style="color: #f8f8f2;">postgres</span>
<span style="color: #f8f8f2;">zfs</span> <span style="color: #66d9ef;">rollback</span> <span style="color: #f8f8f2;">pg_zfs</span><span style="color: #f92672;">@</span><span style="color: #f8f8f2;">blogpost</span>
<span style="color: #f92672;">/</span><span style="color: #f8f8f2;">pg_zfs</span><span style="color: #f92672;">/</span><span style="color: #f8f8f2;">pgsql</span><span style="color: #f92672;">/</span><span style="color: #f8f8f2;">bin</span><span style="color: #f92672;">/</span><span style="color: #f8f8f2;">pg_ctl</span> <span style="color: #f92672;">-</span><span style="color: #f8f8f2;">D</span> <span style="color: #f92672;">/</span><span style="color: #f8f8f2;">pg_zfs</span><span style="color: #f92672;">/</span><span style="color: #f8f8f2;">pgdata</span><span style="color: #f92672;">/</span> <span style="color: #66d9ef;">start</span>
<span style="color: #f8f8f2;">psql</span> <span style="color: #f92672;">-</span><span style="color: #f8f8f2;">q</span> <span style="color: #f92672;">-</span><span style="color: #f8f8f2;">t</span> <span style="color: #f92672;">-</span><span style="color: #f8f8f2;">A</span> <span style="color: #f92672;">-</span><span style="color: #66d9ef;">c</span> <span style="color: #e6db74;">"select count(*) from test_compression"</span> <span style="color: #f8f8f2;">monkey</span>
<span style="color: #ae81ff;">5000000</span>
</pre>
</div>
<br />
<span style="font-family: inherit; font-size: large;">
All Done!</span><br />
<span style="font-family: inherit; font-size: large;">One thing to keep in mind is that all these operations are nearly instant.</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">At <a href="https://omniti.com/">OmniTI</a>, we use ZFS on OmniOS a lot and for this reason we have developed some scripts that can help with automation, sending snapshots on remote servers etc. These scripts can be found <a href="https://github.com/omniti-labs/pgtreats/tree/master/tools/ZFS_Database_Backup_Restore">here</a>.</span><br />
<span style="font-family: inherit; font-size: large;">Biggest con of ZFS is that there is no huge acceptability in linux, people are reporting that open-zfs is production ready, but personally i haven't used it with linux. I've seen that Canonical would promote it but there were some legal issues (?). Thing is that and now, with redhat ditching btrfs it's a good opportunity for ZFS to be used more. So if you have test / development environments , maybe you could give it a try and spread the word of zfs-awesomeness.</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;"><br /></span><br />
<span style="font-family: inherit; font-size: large;">Thanks for reading.</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">Vasilis Ventirozos</span><br />
<span style="font-family: inherit; font-size: large;">OmniTI Computer Consulting</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
</div>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com2tag:blogger.com,1999:blog-826910957631479390.post-56861042156728795392017-06-16T05:01:00.003-07:002018-03-01T08:32:20.627-08:00PostgreSQL on ZFS with BPF tracing on top.<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">At <a href="https://omniti.com/does/data-management">OmniTI</a> we love solaris, my personal favourite features are ZFS and DTrace. Unfortunately not many run postgres on solaris so i have decided to implement similar features in linux. Instead of Dtrace i'll install BPF, in-kernel bytecode that can be used for tracing introduced in recent kernels (4.X). </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">This post will be a part of a three series post. In this post we'll start with setup, in part #2 with ZFS and how to use it for backups / snapshots. In part #3 we'll dig into BPF a bit more.</span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">Step 1 is to setup a new ubuntu. I setup a VM using </span><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><b>ubuntu-16.04.2-server-amd64.iso.</b></span></span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><b><br /></b></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">As root :</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: large;">Add the repo for bcc :</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">> echo "deb [trusted=yes] https://repo.iovisor.org/apt/xenial xenial-nightly main" | sudo tee /etc/apt/sources.list.d/iovisor.list</span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">sudo apt-get update</span></span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">Install all necessary and some optional packages :</span></span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">> apt-get install -y sudo wget apt-transport-https joe less build-essential libreadline-dev \</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">zlib1g-dev flex bison libxml2-dev libxslt-dev libssl-dev openssh-server screen git \</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">binutils bcc bcc-tools libbcc-examples python-bcc zfsutils-linux \</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">systemtap systemtap-sdt-dev</span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">Make sure kernel is properly compiled :</span></span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">> cat /boot/config-`uname -r` |grep BPF</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">CONFIG_BPF=y</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">CONFIG_BPF_SYSCALL=y</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">CONFIG_NETFILTER_XT_MATCH_BPF=m</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">CONFIG_NET_CLS_BPF=m</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">CONFIG_NET_ACT_BPF=m</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">CONFIG_BPF_JIT=y</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">CONFIG_HAVE_BPF_JIT=y</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">CONFIG_BPF_EVENTS=y</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">CONFIG_TEST_BPF=m</span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">Test BCC (stands for BPF Compiler Collection)</span></span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">> python /usr/share/bcc/examples/tracing/bitehist.py</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">Tracing... Hit Ctrl-C to end.</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">^C</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> kbytes : count distribution</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> 0 -> 1 : 7 |************ |</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> 2 -> 3 : 0 | |</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> 4 -> 7 : 22 |****************************************|</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> 8 -> 15 : 19 |********************************** |</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> 16 -> 31 : 8 |************** |</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> 32 -> 63 : 6 |********** |</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> 64 -> 127 : 1 |* |</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> 128 -> 255 : 0 | |</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> 256 -> 511 : 0 | |</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> 512 -> 1023 : 1 |* |</span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">Now its time to install postgres on a zfs partition, in my case i had a disk (sdb) attached on my VM :</span></span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">> fdisk /dev/sdb</span></span><br />
<span style="font-size: large;"><span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Welcome to fdisk (util-linux 2.27.1).</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">Changes will remain in memory only, until you decide to write them.</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">Be careful before using the write command.</span><br />
<span style="font-size: large;"><span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Device does not contain a recognized partition table.</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">Created a new DOS disklabel with disk identifier 0x4226e0bf.</span><br />
<span style="font-size: large;"><span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Command (m for help): n</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">Partition type</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> p primary (0 primary, 0 extended, 4 free)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> e extended (container for logical partitions)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">Select (default p): p</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">Partition number (1-4, default 1):</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">First sector (2048-41943039, default 2048):</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">Last sector, +sectors or +size{K,M,G,T,P} (2048-41943039, default 41943039):</span><br />
<span style="font-size: large;"><span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Created a new partition 1 of type 'Linux' and of size 20 GiB.</span></span><br />
<span style="font-size: large;"><span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Command (m for help): w</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">The partition table has been altered.</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">Calling ioctl() to re-read partition table.</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">Syncing disks.</span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">To create the pool :</span></span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">> sudo zpool create pg_zfs /dev/sdb1</span></span><br />
<span style="font-size: large;"><span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">> zpool status</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> pool: pg_zfs</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> state: ONLINE</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"> scan: none requested</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">config:</span><br />
<span style="font-size: large;"><span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>NAME STATE READ WRITE CKSUM</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"><span class="Apple-tab-span" style="white-space: pre;"> </span>pg_zfs ONLINE 0 0 0</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"><span class="Apple-tab-span" style="white-space: pre;"> </span> sdb1 ONLINE 0 0 0</span><br />
<span style="font-size: large;"><span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">errors: No known data errors</span></span><br />
<span style="font-size: large;"><span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">> mount |grep pg_</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;"><b>pg_zfs on /pg_zfs type zfs</b> (rw,relatime,xattr,noacl)</span><br />
<span style="font-size: large;"><span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">> ls /pg_zfs/ -l</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">total 0</span><br />
<span style="font-size: large;"><span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">> cd /pg_zfs/</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">> mkdir pgsql</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">> mkdir pgdata</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">> chown postgres:postgres pgsql/</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">> chown postgres:postgres pgdata/</span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">Now with everything ready compile postgres from source :</span></span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">> wget -c https://ftp.postgresql.org/pub/source/v10beta1/postgresql-10beta1.tar.gz</span></span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">> tar zxfv postgresql-10beta1.tar.gz</span></span><br />
<span style="font-size: large;"><span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">> cd postgresql-10*</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">> ./configure --prefix=/pg_zfs/pgsql/ <b>--enable-dtrace</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">> make -j 4 world</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">> make -j 4 install-world</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">> export PATH=$PATH:/pg_zfs/pgsql/bin</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">> export LD_LIBRARY_PATH=/pg_zfs/pgsql/lib</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">> export PGDATA=/pg_zfs/pgdata</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">> initdb</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">> pg_ctl start</span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">At this point, postgres binaries and datafiles are on zfs. Now to check the probes we have available :</span></span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">/usr/share/bcc/tools/tplist -l /pg_zfs/pgsql/bin/postgres |awk {'print $2'}</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:clog__checkpoint__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:clog__checkpoint__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:multixact__checkpoint__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:multixact__checkpoint__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:subtrans__checkpoint__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:subtrans__checkpoint__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:twophase__checkpoint__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:twophase__checkpoint__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:transaction__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:transaction__commit</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:transaction__abort</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:wal__buffer__write__dirty__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:wal__buffer__write__dirty__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:wal__switch</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:checkpoint__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:checkpoint__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:wal__insert</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:statement__status</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:buffer__flush__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:buffer__flush__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:buffer__read__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:buffer__read__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:buffer__write__dirty__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:buffer__write__dirty__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:buffer__sync__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:buffer__sync__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:buffer__sync__written</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:buffer__checkpoint__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:buffer__checkpoint__sync__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:buffer__checkpoint__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:lock__wait__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:lock__wait__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:deadlock__found</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:lwlock__wait__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:lwlock__wait__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:lwlock__acquire</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:lwlock__condacquire</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:lwlock__condacquire__fail</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:lwlock__acquire__or__wait</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:lwlock__acquire__or__wait__fail</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:lwlock__release</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:smgr__md__read__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:smgr__md__read__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:smgr__md__write__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:smgr__md__write__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:query__parse__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:query__parse__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:query__rewrite__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:query__rewrite__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:query__plan__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:query__plan__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:query__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:query__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:query__execute__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:query__execute__done</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:sort__start</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">postgresql:sort__done</span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;">To make sure tracing works properly, while running a statement on a different terminal :</span></span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">> sudo /usr/share/bcc/tools/dbslower postgres -p 1208</span></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">Tracing database queries for pids 1208 slower than 1 ms...</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">TIME(s) PID MS QUERY</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: large;">2.729496 1208 2399.665 insert into test select * from generate_series (1,100000);</span><br />
<span style="font-size: large;"><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: medium;"><br /></span></span><br />
<span style="font-family: inherit; font-size: large;">Thanks for reading.</span><br />
<span style="font-family: inherit; font-size: large;"><br /></span>
<span style="font-family: inherit; font-size: large;">Vasilis Ventirozos</span><br />
<span style="font-family: inherit; font-size: large;">OmniTI Computer Consulting</span>Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com3tag:blogger.com,1999:blog-826910957631479390.post-57158947542506645952017-06-10T03:55:00.002-07:002017-06-10T07:58:52.407-07:00An unusual upgrade<span style="font-size: large;">I have mentioned in previous posts that in my 4 years with <a href="https://omniti.com/does/data-management">OmniTI</a>, we've tackled a lot of migrations. Most of them are usually the "typical" procedure. The methodology we use is more or less explained <a href="https://evol-monkey.blogspot.gr/2017/01/postgresql-upgrades-methodology.html">here</a>. Last week we had a usecase for a kind of "unusual" upgrade, a 9.2 compiled with </span><br />
<span style="font-size: large;">"--disable-integer-datetimes" meaning that all datetimes were represented as floating point internally, something that was the default at up to 8.3. This changed at (i think) 8.4 where datetimes were represented as int64 which offers more precision. </span><br />
<span style="font-size: large;">The requirement was to migrate the database to a new one that will use integer datetimes with the minimum possible downtime. Obviously a direct upgrade wouldn't work and pg_dump / restore was not an option so we decided to approach and test this scenario differently.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The general idea is the following :</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Upgrade to a 9.6 that was compiled with "--disable-integer-datetimes" and then using something like <a href="https://www.2ndquadrant.com/en/resources/pglogical/">pglogical</a> or <a href="https://github.com/omniti-labs/mimeo">mimeo</a> to replicate to another 9.6 that would use integer datetimes. For this, i used 2 containers and pagila test database to make this simulation as much realistic as i could. In this post i will describe the i steps I followed.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Installed both 9.2 and 9.6 on the same box :</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">9.2.21 with the following options :</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">./configure --prefix=/home/postgres/pgsql92/ --disable-integer-datetimes</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">make -j 8 world</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">sudo make install-world</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">9.6.2 with the following options :</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">./configure --prefix=/home/postgres/pgsql96/ --disable-integer-datetimes</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">make -j 8 world</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">sudo make install-world</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">initiated a new cluster and started 9.2, loaded pagila testdb (schema and data), started the database.</span><br />
<span style="font-size: large;">From now on this will act like my production database.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">downloaded and installed pglogical 2.0.1 9.6 :</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">wget -c http://packages.2ndquadrant.com/pglogical/tarballs/pglogical-2.0.1.tar.bz2</span><br />
<span style="font-size: large;">uncompress :</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">tar jxfv pglogical-2.0.1.tar.bz2</span><br />
<span style="font-size: large;">compile and install :</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">make USE_PGXS=1 clean all</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">sudo PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/postgres/pgsql96/bin make USE_PGXS=1 install</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Note: i installed 9.6 before upgrading because pglogical requires some changes in the parameters (shown later) and a library loaded, so in order not to restart twice i had it installed before the upgrade.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">initiated a new 9.6 cluster so i can upgrade, stopped 9.2 and upgraded 9.2 to 9.6 :</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">pre upgrade check :</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">pgsql96/bin/pg_upgrade -b /home/postgres/pgsql92/bin/ -B /home/postgres/pgsql96/bin/ -c -d /home/postgres/pgdata92/ -D /home/postgres/pgdata96/ -v</span><br />
<span style="font-size: large;">stopped 9.2 :</span><br />
<span style="font-size: large;">-- outage starts --</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">pgsql92/bin/pg_ctl -D /home/postgres/pgdata92/ stop</span><br />
<span style="font-size: large;">upgrade:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">pgsql96/bin/pg_upgrade -b /home/postgres/pgsql92/bin/ -B /home/postgres/pgsql96/bin/ -d /home/postgres/pgdata92/ -D /home/postgres/pgdata96/ -v -k</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">added the following in postgresql.conf :</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">wal_level = 'logical'</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">max_worker_processes = 10</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">max_replication_slots = 10</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">max_wal_senders = 10</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">shared_preload_libraries = 'pglogical'</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">track_commit_timestamp = on</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">on master hba.conf (and slave, cause why not) added :</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">host replication postgres 10.0.0.1/16 trust </span><br />
<span style="font-size: large;"><i>(security was not a concern so "trust" was ok)</i></span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">started 9.6</span><br />
<span style="font-size: large;">-- outage stops --</span><br />
<span style="font-size: large;">analyzed 9.6 and cleaned up 9.2</span><br />
<span style="font-size: large;">and issued "<span style="font-family: "courier new" , "courier" , monospace;">create extension pg_logical;</span>" to postgres</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">At this point i had my "production" db upgraded to 9.6 with pglogical installed and everything ready for logical replication.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">On the second box that would have postgres 9.6,compiled <b>without</b> the "--disable-integer-datetimes" flag. I installed pglogical with exactly the same way i did for the first box and at this point i was ready to replicate :</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">on production (provider) I created a new node and added a set with all objects in public schema:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">SELECT pglogical.create_node( node_name := 'provider1', dsn := 'host=10.0.0.2 port=5432 dbname=monkey' );</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">SELECT pglogical.replication_set_add_all_tables('default', ARRAY['public']);</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><b><i>Keep in mind :</i></b></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">ERROR: table payment_p2007_01 cannot be added to replication set default</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">DETAIL: table does not have PRIMARY KEY and given replication set is configured to replicate UPDATEs and/or DELETEs</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">HINT: Add a PRIMARY KEY to the table</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">ALL tables that are going to be replicated need to have a primary key.</span><br />
<span style="font-size: large;">after adding a pk to the tables that didn't have one i went to the slave and i did :</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Create a node for the subscriber :</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">SELECT pglogical.create_node(node_name := 'subscriber1',dsn := 'host=10.0.0.3 port=5432 dbname=monkey');</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Started the replication process which sync'd (schema and data): </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">SELECT pglogical.create_subscription( subscription_name := 'subscription1',provider_dsn := 'host=10.0.0.2 port=5432 dbname=monkey', synchronize_structure := true);</span><br />
<br />
<span style="font-size: large;">To verify that these 2 databases have different storage types :</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">postgres@old_server:~/pgdata96$ pg_controldata |grep Date</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Date/time type storage: <b> floating-point numbers</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">postgres@new_server:~/pgdata$ pg_controldata |grep Date</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Date/time type storage: <b>64-bit integers</b></span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The database was transferred and from now on replicated.</span><br />
<span style="font-size: large;">At this point, if this was the real deal it would be preferred to first transfer the schema and then start replication just to be sure that all objects will transfer but in my case i didn't really care about that.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><b><i>Keep in mind that since postgres 10 floating point datetimes are no longer supported. If you tried to compile it it would give :</i></b></span><br />
<span style="font-size: large;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">postgres@a1bdb0750dc5:~/postgresql-10beta1$ ./configure --disable-integer-datetimes</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">checking build system type... x86_64-pc-linux-gnu</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">checking host system type... x86_64-pc-linux-gnu</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">checking which template to use... linux</span><br />
<b><span style="font-family: "courier new" , "courier" , monospace; font-size: medium;">configure: error: --disable-integer-datetimes is no longer supported </span></b><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">This is a pretty specialized scenario, but from what i saw there are some databases out there having their date times stored as floating points that hopefully could benefit from this migration procedure.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Thanks for reading</span><br />
<span style="font-size: large;">-- Vasilis Ventirozos</span><br />
<span style="font-size: large;"><br /></span>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com3tag:blogger.com,1999:blog-826910957631479390.post-44576987228875618002017-06-09T06:39:00.000-07:002017-06-09T06:39:20.175-07:00Tip for faster wal replay on a slave<span style="font-size: large;">I've been in situations where i need a slave db to replay a lot of wal files fast, and by a lot i mean tens of thousands. This could happen because of a reporting database refreshing or simply because a slave was down for an extended period of time. It's known that lowering shared_buffers speeds up wal replay for obvious reasons, but by how much ?</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">I did a benchmark on an old server and the results are interesting :</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">With <b>32GB</b> of shared buffers and with 6390Mb of wals (1840 wal files)</span><br />
<span style="font-size: large;">it took 1408 seconds to complete the replay.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">With <b>64MB</b> of shared buffers and with 6510Mb of wals (1920 wal files)</span><br />
<span style="font-size: large;">it took 1132 seconds to complete the replay.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">My test was done by stopping the slave, inserting 50 mil rows to a test table, wait for the wal transfer to complete, then stop the master and start the slave and watch OmniPITR logs.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">The performance gain in wal replay was about 20% in postgres 10beta1 which doesn't sound bad, especially in times of need.</span><br />
<span style="font-size: large;"><br /></span>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Thanks for reading</span><br />
<span style="font-size: large;">-- Vasilis</span>Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com2tag:blogger.com,1999:blog-826910957631479390.post-37058145501140510992017-05-19T06:37:00.002-07:002017-05-19T06:37:15.688-07:00Reusing an old master (as slave) after a switchover<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">Todays blogpost is old news but it's not very well known and it has to do with how to add an old master after a slave has been promoted. <a class="list" href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=search;h=985bd7d49726c9f178558491d31a570d47340459;s=Fujii+Masao;st=author" style="color: black; text-decoration: none;" title="Search for commits authored by Fujii Masao">Fujii Masao</a> explained the situation in his <a href="https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=985bd7d49726c9f178558491d31a570d47340459">patch</a> back in the day.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">So in todays post i will be demonstrating a combination of replication slots for retaining the writes on a promoted slave and how to re-introduce an old master to the replication.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">Say that we have a master-slave setup, stop the master gracefully and create and activate a replica slot on the slave :</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: large;">monkey=# SELECT * FROM pg_create_physical_replication_slot('this_is_a_replica_slot');</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;"> slot_name | xlog_position</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">------------------------+---------------</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;"> this_is_a_replica_slot |</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">(1 row)</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: large;">postgres@bf9823730feb:~$ pg_receivexlog -D . -S this_is_a_replica_slot -v -h 10.0.0.3 -U repuser</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">pg_receivexlog: starting log streaming at 0/4000000 (timeline 1) ^C</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">pg_receivexlog: received interrupt signal, exiting</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">pg_receivexlog: not renaming "000000010000000000000004.partial", segment is not complete</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">pg_receivexlog: received interrupt signal, exiting</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">This way all the changes after the slave promotion will be retained. Now , lets promote the slave and check the replication slot status.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: large;">postgres@bf9823730feb:~$ pg_ctl promote</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">server promoting</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: large;">postgres@bf9823730feb:~$ psql -c "SELECT slot_name, database,active, pg_xlog_location_diff(pg_current_xlog_insert_location(), restart_lsn) AS retained_bytes FROM pg_replication_slots;" postgres</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;"> slot_name | database | active | retained_bytes</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">------------------------+----------+--------+----------------</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;"> this_is_a_replica_slot | | f | 9056</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">As you can see , the new master is now retaining all writes. Now, to the old master, lets make a recovery.conf that looks like this :</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: large;">standby_mode = 'on'</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">primary_conninfo = 'user=repuser host=10.0.0.3 port=5432 application_name=a_slave'</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">trigger_file = '/home/postgres/pgdata/finish.recovery'</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">primary_slot_name = 'this_is_a_replica_slot'</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">recovery_target_timeline = 'latest'</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">where host is obviously the new master. </span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">Start the old master and you should see something similar to:</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: large;">LOG: entering standby mode</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">LOG: consistent recovery state reached at 0/5000098</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">LOG: database system is ready to accept read only connections</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">LOG: invalid record length at 0/5000098: wanted 24, got 0</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">LOG: fetching timeline history file for timeline 2 from primary server</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">LOG: started streaming WAL from primary at 0/5000000 on timeline 1</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">LOG: replication terminated by primary server</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">DETAIL: End of WAL reached on timeline 1 at 0/5000098.</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">LOG: new target timeline is 2</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">LOG: restarted WAL streaming at 0/5000000 on timeline 2</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: large;">LOG: redo starts at 0/5000098</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">This would also work if you had archiving enabled instead of replication slots. G</span><span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">iven that you have plenty of available disk space on the new master, it should be fine to keep the old master down for any reasonable amount of time and re-introduce it to the replication without issues.</span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><b>The only 2 things that you have to make sure is to shutdown postgres on old-master gracefully and that the slave has caught up before the promotion.</b></span><br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;">Thanks for reading.</span><br />
<br />
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<span style="font-family: Arial, Helvetica, sans-serif; font-size: large;"><br /></span>
<br />Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com0tag:blogger.com,1999:blog-826910957631479390.post-23951392871603966442017-05-09T08:08:00.002-07:002017-05-09T08:32:37.361-07:00PostgreSQL statistics as data<span style="font-size: large;"> Postgres provides a lot of information when it comes to statistics. Only problem is that all the statistics are frozen in time the moment you query the system catalog views. Sometimes you need to see whats happening over time or in the past,and this is easily solvable with a script and a crontab entry. At <a href="https://omniti.com/">OmniTI</a> we use a project called <a href="https://github.com/omniti-labs/system_monitoring">system_monitoring</a> a lot. This is basically a perl scheduler that runs as a deamon exporting useful information from postgres (and more) to tab delimited text files for later investigation. If you have decent one liner skills this is very helpful: you can answer questions about what happened yesterday night, what was running, who locked who etc. It's very good but data analysis can be a pain in the ass sometimes, especially if you need to reimport to a db and analyze with SQL. So I decided to approach this in a different way, by storing the statistics to a schema. Thats why i wrote <a href="https://github.com/vventirozos/statz">statz</a>. </span><br />
<span style="font-size: large;">Statz is a very simple python script that will gather all postgres statistics for a period of time in an interval, aggregate them and keep raw and aggregated data in its own schema. The raw data that it keeps are just a <b>now(),*</b> from the most important statistics tables :</span><br />
<br />
<ul>
<li><span style="font-size: large;">pg_stat_user_indexes</span></li>
<li><span style="font-size: large;">pg_stat_activity</span></li>
<li><span style="font-size: large;">pg_locks</span></li>
<li><span style="font-size: large;">pg_stat_user_tables</span></li>
<li><span style="font-size: large;">pg_stat_database</span></li>
<li><span style="font-size: large;">pg_stat_bgwriter</span></li>
</ul>
<div>
<span style="font-size: large;">But its relatively easy to add more, even application specific data.</span></div>
<h4>
<span style="font-size: large;">How to use :</span></h4>
<div>
<span style="font-size: large;">statz needs a database to connect (for now it's the same with the database that monitors), an interval and a total duration (<u>interval and total duration should be in seconds</u>).</span></div>
<div>
<span style="font-size: large;">So, something like:</span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;"><b>./statz.py -c dbname=monkey -i 5 -t 120</b> </span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">Would </span><span style="font-size: large;">initialize a schema called </span><b>statz </b><span style="font-size: large;">(this for now is hardcoded)</span><span style="font-size: large;"> , </span></div>
<div>
<span style="font-size: large;"><b><u>if schema exists it will drop it (cascade) and recreate it.</u></b></span></div>
<div>
<span style="font-size: large;">In a loop that will last for <total time> , sleeping for <interval> it will keep output of select now(),* from the tables listed above, populating the following tables :</span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<b><span style="font-family: "courier new" , "courier" , monospace; font-size: large;">statz.backend_activity</span></b></div>
<div>
<b><span style="font-family: "courier new" , "courier" , monospace; font-size: large;">statz.bgwriter_activity</span></b></div>
<div>
<b><span style="font-family: "courier new" , "courier" , monospace; font-size: large;">statz.database_activity</span></b></div>
<div>
<b><span style="font-family: "courier new" , "courier" , monospace; font-size: large;">statz.index_activity</span></b></div>
<div>
<b><span style="font-family: "courier new" , "courier" , monospace; font-size: large;">statz.lock_activity</span></b></div>
<div>
<b><span style="font-family: "courier new" , "courier" , monospace; font-size: large;">statz.table_activity</span></b></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">In the same loop it will aggregate and populate the following aggregated tables.</span></div>
<div>
<b><span style="font-family: "courier new" , "courier" , monospace; font-size: large;">statz.database_activity_agg</span></b></div>
<div>
<b><span style="font-family: "courier new" , "courier" , monospace; font-size: large;">statz.table_activity_agg</span></b></div>
<br />
<br />
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">The aggregated tables look like this :</span></div>
<div>
<span style="font-size: large;">for the database:</span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<br />
<div>
<span style="font-family: "courier new" , "courier" , monospace;">snap_date | 2017-05-08 16:35:17</span></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">interval | 00:00:02</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">step | 00:00:10</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">datname | monkey</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">commits | 863</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_returned | 21761</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_fetched | 9102</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_inserted | 1662</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_updated | 2251</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_deleted | 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">blocks_read | 89</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">blocks_hit_cached | 57572</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">commits_per_sec | 432</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_returned_per_sec | 10881</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_fetched_per_sec | 4551</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_inserted_per_sec | 831</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_updated_per_sec | 1126</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_deleted_per_sec | 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">blocks_read_per_sec | 45</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">blocks_hit_cached_per_sec | 28786</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">txn_per_sec | 432</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">cache_hit_ratio | 99.8456</span></div>
</div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">and for each table :</span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">snap_date | 2017-05-08 16:35:49</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">interval | 00:00:02</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">step | 00:00:42</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">table_name | public.pgbench_tellers</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">seq_scans | 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">seq_rows_read | 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">index_scans | 5807</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">index_rows_fetched | 5807</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_inserted | 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_updated | 5807</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_deleted | 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_hot_updated | 5759</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">live_row_count | 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">n_dead_tup | 634</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">seq_scans_per_sec | 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">seq_rows_read_per_sec | 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">index_scans_per_sec | 5807</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">index_rows_fetched_per_sec | 2904</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_inserted_per_sec | 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_updated_per_sec | 2904</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_deleted_per_sec | 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">rows_hot_updated_per_sec | 2880</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">live_row_count_per_sec | 0</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">n_dead_tup_per_sec | 317</span></div>
</div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<br /></div>
<div>
<span style="font-size: large;">I have also included a couple of views that are useful for fast monitoring :</span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<div>
<b><span style="font-size: large;">statz.db_stats_per_sec </span></b></div>
<div>
<b><span style="font-size: large;">statz.table_stats_per_sec</span></b></div>
</div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">Keep in mind that the numbers are being calculated using the last inserted row and since its in the same loop with populating the historical tables <b>the information will be refreshed every <interval> </b></span></div>
<div>
<span style="font-size: large;">Which makes this a great tool for things like :</span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<i><span style="font-size: large;">watch -n 2 "psql -x -c 'select * from statz.database_activity_agg order by step desc limit 1;' monkey"</span></i></div>
<div>
<i><span style="font-size: large;"><br /></span></i></div>
<div>
<span style="font-size: large;">You can even create and send graphs like :</span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeKZNK1cn9PSVluQgorNtTkZNu3b2-_iUmscqrSAeamUjN_rjOeMzTe-2Ln9LNuvqhdKNNRygq8uE-odk3GLFurw4qJbqkGlDIgTh8IdBPtnOvB4RRWtXm7De6w_UR2wmrLEtRh6g3HMDK/s1600/Screen+Shot+2017-05-09+at+15.59.26.png" imageanchor="1"><img border="0" height="235" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjeKZNK1cn9PSVluQgorNtTkZNu3b2-_iUmscqrSAeamUjN_rjOeMzTe-2Ln9LNuvqhdKNNRygq8uE-odk3GLFurw4qJbqkGlDIgTh8IdBPtnOvB4RRWtXm7De6w_UR2wmrLEtRh6g3HMDK/s400/Screen+Shot+2017-05-09+at+15.59.26.png" width="400" /></a></div>
<div>
<br /></div>
<h4>
<span style="font-size: large;">Usecases</span></h4>
<span style="font-size: large;"><br /></span>
<span style="font-size: large;">Statz is gathering a lot of data so it's not meant to run 24/7. Thats why it has an interval and a total time and <b>thats why it will destroy old statistical data when re-run</b>. </span><br />
<span style="font-size: large;">Say that you observed that during night, queries are getting slower or wal file generation is increased or any abnormality (locks / unknown statements / rollbacks etc). You can schedule statz to run for this period of time and analyze data next morning. Keeping snapshots of pg_stat_activity is also helpful for slow query investigation where you can do things like :</span><br />
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<b><span style="font-size: large;">select (snap_date - xact_start) as age,user,query from backend_activity order by 1 desc;</span></b></div>
<div>
<br /></div>
<div>
<span style="font-size: large;">This script is also useful for analyzing application specific usage patterns, To get statistics from benchmarks and fine tune checkpoints , missing indexes etc..</span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">And because it all happens in a single transaction, <b>snap_date</b> can be used as a key linking all statz tables.</span></div>
<div>
<span style="font-size: large;">This means that if you see database doing a lot of commits for example, you can use <b>snap_date</b> to find the busy table in <b>statz.table_activity</b> or you can check locks or statements run at that specific time in the rest of the tables.</span></div>
<div>
<span style="font-size: large;"><br /></span></div>
<div>
<span style="font-size: large;">link for statz can be found <a href="https://github.com/vventirozos/statz">here</a>.</span></div>
<h4>
<span style="font-size: large;">Next steps</span></h4>
<div>
<span style="font-size: large;">I plan on the following things :</span></div>
<div>
<ul>
<li><span style="font-size: large;">Refactor schema / statements for performance.</span></li>
<li><span style="font-size: large;">More aggregating statements for the rest of the tables.</span></li>
<li><span style="font-size: large;">Possibly (live) graphs or maybe a dashboard.</span></li>
<li><span style="font-size: large;">Possibly rewrite some parts for better general use.</span></li>
<li><span style="font-size: large;">Lately im reading a book about machine learning in python i might experiment with that a bit.</span></li>
</ul>
<div>
<span style="font-size: large;"><br /></span></div>
</div>
<div>
<span style="font-size: large;">Thanks for reading.</span></div>
<div>
<br /></div>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com0tag:blogger.com,1999:blog-826910957631479390.post-10477721193732095122017-01-25T09:30:00.000-08:002017-05-09T04:20:02.505-07:00PostgreSQL upgrades - Methodology<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> Yesterday, i came across a conversation in postgres slack channel, someone was interested in upgrading his database and i realised that there is a pattern i see all the time. Those who are interested in upgrading major postgres versions, only ask questions about pg_upgrade. Don't get me wrong, pg_upgrade is awesome its well <a href="https://www.postgresql.org/docs/9.6/static/pgupgrade.html">documented</a>, and it explains things much better than i can. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">During my time in <a href="https://omniti.com/">OmniTI</a> we've done many... MANY upgrades some of them pretty challenging,</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">so i have come to realise that the upgrade tool is not as important as the plan and the methodology, especially during an upgrade or a migration. For this reason this post won't be about how to use pg_upgrade, instead it will be all about methodology, minimising downtime and reducing risk to almost zero.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">For this post i will be showing the steps i would follow in order to upgrade an 8.4.22 to 9.6.1. I will be using docker containers using a <a href="https://github.com/vventirozos/pgdocker/blob/master/Dockerfile">dockerfile</a> that i wrote for lab use (more info about this docker file can be found in <a href="http://evol-monkey.blogspot.gr/2016/12/custom-postgres-installation-with-docker.html">this post</a>) I will include a couple of "traps" that could potentially cause trouble to my upgrade just so i can show how to work around them.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">A typical postgres installation would have one master and one or more slaves and maybe a dev/test server. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">For the needs of this post , i will concentrate on the master and on the test. As a matter of fact the test server will be much more important because thats where we will develop an upgrade plan.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">So, we need 2 containers :</span><br />
<u><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">An upgrade target host</span></u><br />
<u><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">An upgrade plan development server (aka a test machine)</span></u><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<span style="font-family: inherit; font-size: xx-small;">So , lets assume that we have a master with the following objects that we want to upgrade :</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">postgres=# select version();</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> version</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">---------------------------------------------------------------------------------------------------------------------------</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> PostgreSQL 8.4.22 on x86_64-unknown-linux-gnu, compiled by GCC gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609, 64-bit</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">(1 row)</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">postgres=# \dt</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> List of relations</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> Schema | Name | Type | Owner</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">--------+------------------------+-------+----------</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> public | another_small_table | table | postgres</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> public | big_reporting_table | table | postgres</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> public | really_important_table | table | postgres</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> public | small_oltp_table | table | postgres</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">(4 rows)</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">postgres=# \dv</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> List of relations</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> Schema | Name | Type | Owner</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">--------+------------------+------+----------</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> public | my_stat_activity | view | postgres</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">(1 row)</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<span style="font-family: inherit; font-size: xx-small;">First order of business would be to create a replica (or to dump and restore if size permits) on the test server that should run on exactly the same postgres version. </span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: xx-small;">Next, you should check if the test server has the same configuration and if </span><b><span style="font-size: xx-small;">all contrib modules</span></b><span style="font-size: xx-small;"> are installed (not calling them extensions because ... 8.4.x).</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">So compare postgresql.conf on these 2 servers and compare output of <i>ls -l <postgres binary installation directory>/lib. </i></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">In my case i didn't have pg_buffercache , so i installed it. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">At this point it is a good idea to write down the extensions you would need for 9.6, in my case just pg_buffercache.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<span style="font-family: inherit; font-size: xx-small;">To sum up, the steps are :</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<b><span style="font-family: inherit; font-size: xx-small;">On the test server :</span></b></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<ol>
<li><b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Install exactly the same version as production</span></b></li>
<li><b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Transfer the configs from production</span></b></li>
<li><b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Install all the extensions that exist in production</span></b></li>
<li><b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Create a replica from production</span></b></li>
<li><b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Promote the replica to be a standalone master</span></b></li>
</ol>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<span style="font-family: inherit; font-size: xx-small;">At this point we should have a test environment that looks like production. From this moment on, everything should be timed and documented.</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<b><u><span style="font-family: inherit; font-size: xx-small;">On the Test server</span></u></b></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<ol>
<li><b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Install the version you want to upgrade to, in my case 9.6.1 and don't put its binaries in the path (yet) while documenting and timing all the steps. <u>DO NOT overwrite your old binaries</u>.</span></b></li>
<li><b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Port your configuration file to the new version, <u>DO NOT copy the old config file</u>. <b style="font-family: inherit;">Its not just that some parameters have changed, or that some parameters are not compatible from an old to a new pg version, there are new settings commented out that are important and new defaults that you should tune before putting it to production.</b></span></b></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"><b>Initdb a cluster using 9.6 binaries and test your configuration files, remember to <u>start it on a different port than 8.4</u>. </b><b style="text-decoration: underline;">Keep the config files somewhere safe ideally at the same place where you develop your migration plan</b><b>. </b></span></li>
<li><b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Stop 8.4 and 9.6 clusters</span></b></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"><b>Delete the previously created (for config test) cluster and re-initialize a new $PGDATA directory </b><b>using the 9.6 binary. <u>This will be your upgraded $PGDATA directory</u></b></span></li>
<li><b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Test pg_upgrade (9.6 binary) like this : </span></b></li>
</ol>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">pg_upgrade -b ~/pgsql8/bin/ -B ~/pgsql9/bin/ -d ~/pgdata8 -D ~/pgdata9 -c</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Performing Consistency Checks</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">-----------------------------</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking cluster versions ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking database user is the install user ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking database connection settings ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for prepared transactions ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for reg* system OID user data types ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for contrib/isn with bigint-passing mismatch ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for roles starting with 'pg_' ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for invalid "line" user columns ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for large objects ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for presence of required libraries ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking database user is the install user ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for prepared transactions ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">*Clusters are compatible*</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><span style="font-family: inherit; font-size: xx-small;"><br /></span></b>
<span style="font-family: inherit; font-size: xx-small;">everything looks good, lets try to upgrade :</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<b><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">time pg_upgrade -b ~/pgsql8/bin/ -B ~/pgsql9/bin/ -d ~/pgdata8 -D ~/pgdata9 -k</span></b></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Performing Consistency Checks</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">-----------------------------</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking cluster versions ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking database user is the install user ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking database connection settings ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for prepared transactions ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for reg* system OID user data types ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for contrib/isn with bigint-passing mismatch ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for roles starting with 'pg_' ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for invalid "line" user columns ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for large objects ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Creating dump of global objects ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Creating dump of database schemas</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for presence of required libraries ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking database user is the install user ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Checking for prepared transactions ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">If pg_upgrade fails after this point, you must re-initdb the</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">new cluster before continuing.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Performing Upgrade</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">------------------</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Analyzing all rows in the new cluster ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Freezing all rows on the new cluster ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Deleting files from new pg_clog ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Copying old pg_clog to new server ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Setting next transaction ID and epoch for new cluster ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Deleting files from new pg_multixact/offsets ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Setting oldest multixact ID on new cluster ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Resetting WAL archives ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Setting frozenxid and minmxid counters in new cluster ok</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Restoring global objects in the new cluster ok</span><br />
<b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Restoring database schemas in the new cluster</span></b><br />
<b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> postgres</span></b><br />
<b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">*failure*</span></b><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span></b>
<b><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Consult the last few lines of "pg_upgrade_dump_11564.log" for</span></b></span><br />
<b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">the probable cause of the failure.</span></b><br />
<b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Failure, exiting</span></b><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">real<span class="Apple-tab-span" style="white-space: pre;"> </span>0m7.011s</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">user<span class="Apple-tab-span" style="white-space: pre;"> </span>0m0.000s</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">sys<span class="Apple-tab-span" style="white-space: pre;"> </span>0m0.050s</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<span style="font-family: inherit; font-size: xx-small;">This is why we HAVE to test and time things in a test server. If we check the log we'll see :</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">pg_restore: [archiver (db)] Error from TOC entry 142; 1259 16390 <b>VIEW my_stat_activity</b> postgres</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">pg_restore: [archiver (db)] could not execute query: ERROR: <b>column pg_stat_activity.procpid does not exist</b></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">LINE 14: ...CT (now() - pg_stat_activity.query_start) AS age, pg_stat_ac...</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span><span style="font-family: inherit; font-size: xx-small;">This was an intended to happen error, just to show things can go wrong, and that you don't want these kinds of things to happen in production. Its just a view that uses columns from pg_stat_activity that were renamed later on. Same thing can happen from some old extensions (like tsearch) so one more reason to test.</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<span style="font-family: inherit; font-size: xx-small;">In order to fix this i would get the view definition, i would adjust it to work in 9.6, i would<b> test it </b>and when i felt confident that the view works I would write a script that would look s</span><span style="font-family: inherit; font-size: xx-small;">omething like :</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"><b>pg_ctl -D ~/pgdata8 start && psql -c "drop view my_stat_activity" postgres && pg_ctl -D ~/pgdata8 stop && rm -rf ~/pgdata9/* && ~/pgsql9/bin/initdb -D ~/pgdata9/ && time pg_upgrade -b ~/pgsql8/bin/ -B ~/pgsql/bin/ -d ~/pgdata -D ~/pgdata9 -k </b><b>&& cp ~/configs9/*.conf ~/pgdata9/ && </b><b>~/pgsql9/bin/pg_ctl -D ~/pgdata9/ start && psql -c "create view my_stat_activity as select now()-query_start as age,pid,state, query from pg_stat_activity where query !='<IDLE>' order by 1 desc;" postgres && psql -c "create extension pg_buffercache" postgres</b></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<span style="font-family: inherit; font-size: xx-small;">Don't pay too much attention to the code above since its not even tested, just keep a similar mindset that its a really good idea to automate / script everything, scripts are easy to test, easy to predict and easy to keep and change.</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">The steps for the code above would be :</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">start pg8</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">drop problematic view</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">stop pg8</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">empty pg9 data dir</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">initdb pg9 data dir</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">upgrade (this should be timed)</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">copy configs to pg9 config</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">start pg9</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">create view</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">create extensions</span></li>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<span style="font-family: inherit; font-size: xx-small;">Upgrade is not done though. Next step would be to analyze the database and this can take a long time!</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;">The last 2 steps would be to delete old cluster and analyze the new one (because o</span><span style="font-family: inherit; font-size: xx-small;">ptimizer statistics are not kept by pg_upgrade). pg_upgrade provides 2 scripts that could be used for these steps :</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">analyze_new_cluster.sh</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">delete_old_cluster.sh</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<span style="font-family: inherit; font-size: xx-small;">delete_old_cluster.sh can be run now, but before you run analyze_new_cluster you could prioritise work and maybe even run analyze commands in parallel. My dummy database has the following tables :</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<span style="font-family: inherit; font-size: xx-small;"> </span><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> List of relations</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> Schema | Name | Type | Owner</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">--------+------------------------+-------+----------</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> public | another_small_table | table | postgres</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> public | big_reporting_table | table | postgres</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> public | really_important_table | table | postgres</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"> public | small_oltp_table | table | postgres</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<span style="font-family: inherit; font-size: xx-small;">At this point i could say that i want really_important_table analyzed asap followed by another_small_table and small_oltp_table. big_reporting_table can be last or run on a different process. My suggestion is to prioritise important tables first by creating one or more scripts. </span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Remember that we are still working on test server so test and time these analyze scripts. eg :</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: xx-small;">time psql -c "analyze </span><span style="font-size: xx-small;">really_important_table;" postgres</span></span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: xx-small;">time psql -c "analyze </span><span style="font-size: xx-small;">small_oltp_table;</span><span style="font-size: xx-small;">" postgres</span></span><br />
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: xx-small;">time psql -c "analyze </span><span style="font-size: xx-small;">another_small_table;</span><span style="font-size: xx-small;">" postgres</span></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: xx-small;">time psql -c "analyze </span><span style="font-size: xx-small;">big_reporting_table;</span><span style="font-size: xx-small;">" postgres</span></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">In my opinion some of the most important things in an upgrade are :</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Minimising risk of losing data</span></li>
<li><span style="font-size: xx-small;"><span style="font-family: "arial" , "helvetica" , sans-serif;">Minimising downtime</span></span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Providing accurate estimations on downtime to whoever needs to know.</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Providing accurate estimations on analyze. Assuming you'll use hard links (-k) This is the step that will take the most, and during the time that the database lacks statistics, performance will be bad.<b> (READ HOW -k WORKS BEFORE USING IT)</b></span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Not having any surprises.</span></li>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;">I can't emphasis enough on the importance of scripting, timing and testing any upgrade procedure. In my opinion it's more important that the production upgrade itself. </span><span style="font-family: inherit; font-size: xx-small;">Do these steps again and again till you are confident that your procedure :</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Has been tested well enough</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Has been documented well enough</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Can't go wrong</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Is so easy that you could even train your dog to do it.</span></li>
</ul>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;"><b><u>This upgraded test database should now be used to thoroughly test the application against the new pg version.</u></b></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">When you start upgrading production, its a good idea to keep two things in mind</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<br />
<ul>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">Keep a backup handy. If the database is large, this is not as easy as it sounds and restoration time can be a lot, if you can afford the safety of a backup though, do it !!</span></li>
<li><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">ALWAYS have a rollback plan. Make sure a replica of the master is up that has no replication lag, bring it down at the time of the upgrade and be ready to move the application there, just in case something goes south. Even if you don't have a replica in your stack, the upgrade is a good opportunity to get one, even temporarily until the upgrade is done. (protip: if you failover your application to a slave keep connections on your failed-to-upgrade-master restricted making sure no one will commit any transaction).</span></li>
</ul>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-family: inherit; font-size: xx-small;"><br /></span>
<span style="font-family: inherit; font-size: xx-small;">Thanks for reading.</span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: xx-small;">-- Vasilis</span>Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com1tag:blogger.com,1999:blog-826910957631479390.post-64214600603083482762016-12-20T11:34:00.000-08:002016-12-20T11:41:23.714-08:00Traveling in time(lines) with pg_rewind<span style="font-family: "arial" , "helvetica" , sans-serif;">At 9.5 </span><a href="https://www.postgresql.org/docs/9.5/static/app-pgrewind.html"><span style="font-family: "courier new" , "courier" , monospace;">pg_rewind</span></a><span style="font-family: "arial" , "helvetica" , sans-serif;"> was introduced, with this feature it was possible to</span><span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif;"> make a server that is no longer master to follow a promoted standby that has a new timeline. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white;"><span style="color: #333333;">There are at least a few use cases that could benefit from this feature, to name a few :</span></span></span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white; color: #333333;"><br /></span></span>
<span style="color: #333333;"><span style="background-color: white;"><b style="font-family: Arial, Helvetica, sans-serif;">Testing Failovers</b><span style="font-family: "arial" , "helvetica" , sans-serif;"> : Its pretty common for one of customers to ask for a failover simulation just to make sure that everything works, by using </span><span style="font-family: "courier new" , "courier" , monospace;">pg_rewind</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> this is much easier and much faster because we don't have to re-sync the whole database that sometimes is multi-TB sized. </span></span></span><br />
<span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white;"><br /></span></span>
<span style="color: #333333;"><span style="background-color: white;"><b style="font-family: Arial, Helvetica, sans-serif;">Minor version upgrades</b><span style="font-family: "arial" , "helvetica" , sans-serif;"> : during an upgrade you have to put the new binaries in place and restart the database. By using </span><span style="font-family: "courier new" , "courier" , monospace;">pg_rewind</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> if the application is flexible enough , you could upgrade the slave , switchover , upgrade the master and </span><span style="font-family: "courier new" , "courier" , monospace;">pg_rewind</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> the old master to follow the new slave, possibly minimising even more the downtime.</span></span></span><br />
<span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white;"><br /></span></span>
<span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white;"><b>Requirements.</b></span></span><br />
<span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white;"><br /></span></span>
<span style="color: #333333;"><span style="background-color: white;"><span style="font-family: "courier new" , "courier" , monospace;">pg_rewind</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> has a couple of requirements :</span></span></span><br />
<span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white;"><br /></span></span>
<br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">1. data page checksums enabled (</span><span style="font-family: "courier new" , "courier" , monospace;">initdb -k</span><span style="font-family: "arial" , "helvetica" , sans-serif;">)</span><br />
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif;">or</span><br />
<div>
<span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif;">2. parameter </span><code style="background-color: #f9f9f9; border: 0px; color: #333333; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline; white-space: pre-wrap; word-wrap: break-word;"><span style="font-family: "courier new" , "courier" , monospace;">wal_log_hints</span></code><span style="background-color: white; color: #333333; font-family: "arial" , "helvetica" , sans-serif;"> has to be enabled </span><br />
<span style="background-color: white; color: #333333;"><span style="font-family: "arial" , "helvetica" , sans-serif;">3. The old-master (the one we want to re-sync) to be properly shutdown or you'll get: </span></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">target server must be shut down cleanly </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">Failure, exiting</span><br />
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b>How does it work ?</b></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><b><br /></b></span></div>
<div>
<div style="border: 0px; margin-bottom: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: #333333; font-family: "courier new" , "courier" , monospace;">pg_rewind</span><span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"> searches the old-master’s data</span><span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"> directory, finds the data blocks changed during the switchover and then copies only the nesessary blocks from the promoted slave. Keep in mind</span><span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"> that the configuration files will also be copied from the new master so take a look for mis-configurations before you start it ! Also you have</span><span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"> </span><span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;">to have all the</span><span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"> wal files</span><em style="border: 0px; color: #333333; font-family: Arial, Helvetica, sans-serif; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;"> since the last checkpoint </em><span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"> from the old master. Changes are identified by comparing the state of the data blocks present in the data directory</span><span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;"> with the changes logged in the wal files. When the deferrable blocks are identified, the wals are replayed.</span></div>
<div style="border: 0px; margin-bottom: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<span style="color: #333333; font-family: "arial" , "helvetica" , sans-serif;">For more info refer to the documentation <a href="https://www.postgresql.org/docs/9.5/static/app-pgrewind.html">here</a></span></div>
<div style="border: 0px; color: #333333; margin-bottom: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div style="border: 0px; color: #333333; font-family: 'Open Sans', Helvetica, Arial, sans-serif; font-size: 14px; margin-bottom: 1em; outline: 0px; padding: 0px; vertical-align: baseline;">
<b style="background-color: white; font-family: Arial, Helvetica, sans-serif;">How to do it</b></div>
</div>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white; color: #333333;"><br /></span></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white;"><span style="color: #333333;">Recently i've posted about how to use Docker to setup a lab environment, the link can be found <a href="https://evol-monkey.blogspot.com/2016/12/custom-postgres-installation-with-docker.html">here</a>. I'm going to use two containers for this blogpost using the image created from the <a href="https://github.com/vventirozos/pgdocker">dockerfile</a> that i have. Keep in mind that i</span></span>n my Dockerfile, i make sure that </span><span style="font-family: "courier new" , "courier" , monospace;">initdb</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> is run with -k option, that all the parameters are properly set for replication and that archiving is enabled but not doing anything : </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="background-color: white;"><span style="color: #333333;"><br /></span></span></span>
<span style="font-family: "courier new" , "courier" , monospace;">psql -c "show archive_command";</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> archive_command</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">-----------------</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> /bin/true</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">(1 row)</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">before i setup my slave, i create a wal_archive directory and i change the archive_command to simply archive there :</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">mkdir ~/wal_archive</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">psql -c "alter system set archive_command to 'cp %p /home/postgres/wal_archive/%f' ;" postgres</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">ALTER SYSTEM</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">psql -c "select pg_reload_conf() " postgres</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> pg_reload_conf</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">----------------</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> t</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">(1 row)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">psql -c "select pg_switch_xlog() ;" postgres</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> pg_switch_xlog</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">----------------</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> 0/7000078</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">(1 row)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">psql -c "checkpoint;" postgres</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">CHECKPOINT</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">now , from the slave :</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">mkdir ~/wal_archive ; ~/mk_replica.sh</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">waiting for server to shut down.... done</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">server stopped</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">29284/29284 kB (100%), 1/1 tablespace</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">server starting</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">LOG: redirecting log output to logging collector process</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">HINT: Future log output will appear in directory "pg_log".</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">-[ RECORD 1 ]----+--------</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">lag_bytes | 0</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">pid | 106</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">application_name | a_slave</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Next , we promote the slave :</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">pg_ctl promote ; sleep 5 ; psql -c "select pg_is_in_recovery();" postgres</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">server promoting</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">pg_is_in_recovery</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">-------------------</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> f</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">(1 row)</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">And now, time to pg_rewind, from the old master :</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">pg_ctl stop</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">waiting for server to shut down.... done</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">server stopped</span><br />
<b><span style="font-family: "courier new" , "courier" , monospace;">pg_rewind --target-pgdata=/home/postgres/pgdata --source-server="host=10.0.0.3 port=5432 user=postgres dbname=postgres"</span></b><br />
<b><span style="font-family: "courier new" , "courier" , monospace;"><br /></span></b>
<span style="font-family: "arial" , "helvetica" , sans-serif;">servers diverged at WAL position 0/F000098 on timeline 1</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">rewinding from last common checkpoint at 0/F000028 on timeline 1</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">Done!</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">put a recovery.conf on the rewinded master (now slave) and start it up, check that it got connected to the new master and you should be all good.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">postgres=# </span><span style="font-family: "courier new" , "courier" , monospace;">SELECT pg_xlog_location_diff(pg_current_xlog_insert_location(), flush_location) AS lag_bytes,pid, application_name FROM pg_stat_replication;</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> lag_bytes | pid | application_name</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">-----------+-----+------------------</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> 0 | 139 | old_master</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Most of the production database clusters that i've come across don't use database checksums </span><span style="font-family: "arial" , "helvetica" , sans-serif;">which is understandable , because of the performance penalty that checksums have, </span><span style="font-family: "arial" , "helvetica" , sans-serif;">(as an alternative to checksums, in </span><a href="https://omniti.com/" style="font-family: Arial, Helvetica, sans-serif;">OmniTI</a><span style="font-family: "arial" , "helvetica" , sans-serif;"> we use ZFS on </span><a href="http://omnios.omniti.com/" style="font-family: Arial, Helvetica, sans-serif;">OmniOS</a><span style="font-family: "arial" , "helvetica" , sans-serif;"> A LOT, and we love it) </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"> still </span><span style="font-family: "courier new" , "courier" , monospace;">pg_rewind</span><span style="font-family: "arial" , "helvetica" , sans-serif;"> is a nice feature , maybe its not for every database but for those who will use it, it might save a ton of time of waiting to resync an old master.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif;">Thanks for reading</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif;">-- Vasilis Ventirozos</span><br />
</div>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com3tag:blogger.com,1999:blog-826910957631479390.post-74018623550241268892016-12-14T06:54:00.000-08:002016-12-14T06:54:22.290-08:00Adding columns with default value fastI recently came across a discussion in #postgresql channel about how to add a new column that will represent record insert timestamps to a table with minimal locking.<br />
<br />
Normally you would do something like this :<br />
<br />
monkey=# <b>alter table alter_test add column insert_timestamp timestamp without time zone default now();</b><br />
ALTER TABLE<br />
Time: <b><u>13643.983 ms</u></b><br />
monkey=#<b> create index concurrently insert_timestamp_idx on alter_test (insert_timestamp) where insert_timestamp is not null;</b><br />
CREATE INDEX<br />
Time: 1616.108 ms<br />
<br />
This though , would require an AccessExclusiveLock on the table and the application would have to wait for the duration of alter.<br />
What you could do instead is :<br />
<br />
monkey=#<b> alter table alter_test add column insert_timestamp timestamp without time zone;</b><br />
ALTER TABLE<br />
Time: <b><u>1.135 ms</u></b><br />
monkey=# <b>alter table alter_test alter column insert_timestamp set default now();</b><br />
ALTER TABLE<br />
Time: <b><u>1.238 ms</u></b><br />
monkey=# create index concurrently insert_timestamp_idx on alter_test (insert_timestamp) where insert_timestamp is not null;<br />
CREATE INDEX<br />
Time: 1618.788 ms<br />
<br />
As you can see the timings are completely different, in the first case the database has to change data in all pre-existing rows, which isn't really needed since the value will be dummy (it will default to the transaction time).<br />
The second way will also require an AccessExclusiveLock but it wont have to change any data ignoring the old rows so the lock should be almost instant (assuming no blockers exist),<br />
next step you alter the table adding the default value and from now on the new column will have the default value. If you don't want to have nulls, or you want to add a not null constraint to the column, you can update the old (null) rows in a more progressive and non intrusive way later on.<br />
<br />
Thanks for reading<br />
- Vasilis VentirozosEvol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com3tag:blogger.com,1999:blog-826910957631479390.post-24848387861448750152016-12-06T08:53:00.000-08:002016-12-06T08:53:32.729-08:00Custom Postgres installation with Docker<div dir="ltr" style="text-align: left;" trbidi="on">
<div>
<br /></div>
Lately, i wanted to make a new test / lab environment that would be dispensable, fast to deploy and easily customisable. VM's are all nice for lab use and they served me well over the years, but they tend to be slow(er) to deploy, heavy on resources and lets face it,<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTi_WoczUrsExPn0I-3Pzgpq0QazI44WCZawvU5apAxRfmsoDnRP8vN5e0s4rd3X21Su8WSgOYwPc9bFrV2KDtkOQMrS14X_Qc4B8vgDClXyBN1ZSzMC5ChHaTVyeUGFirHyT7Th092u4O/s1600/so+hot.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="256" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTi_WoczUrsExPn0I-3Pzgpq0QazI44WCZawvU5apAxRfmsoDnRP8vN5e0s4rd3X21Su8WSgOYwPc9bFrV2KDtkOQMrS14X_Qc4B8vgDClXyBN1ZSzMC5ChHaTVyeUGFirHyT7Th092u4O/s320/so+hot.jpg" width="320" /></a></div>
<br />
<br />
So i've decided to play around with Docker. I've played with Docker in the past but i haven't done anything too serious, i just took the official postgres Dockerfile and run it to see how it works. This is how i started my whole Docker exploration, i took the official postgres docker file that can be found <a href="https://hub.docker.com/_/postgres/">here</a> and made an image. Immediately i saw that it wasn't exactly what i wanted, i wanted something less generic. The things that i would like are :<br />
<br />
<ul style="text-align: left;">
<li>latest postgres version, compiled from source</li>
<li>easily customisable pgdata and prefix</li>
<li>custom postgresql.conf settings (mostly having it replication ready)</li>
<li>custom shell environment</li>
<li>some extensions</li>
<li>data page checksums</li>
<li>easy replica deployment</li>
</ul>
<div>
So i started creating my own Dockerfile that would fill my needs and at the same time i would learn some more about Docker, 2 birds one stone kind of thing. After several hours and some testing i came up with something that was working (link at the bottom). It still needs some work to be done but i plan maintaining it for my own use so feel free to use it if you like what i've done there.</div>
<div>
<br /></div>
<div>
Let me explain a bit how it works.</div>
<div>
With :</div>
<div>
ENV PGBINDIR=/home/postgres/pgsql</div>
<div>
ENV PGDATADIR=/home/postgres/pgdata</div>
I can customise the installation and the pgdata directories, I install all packages i need, note that <b>comments on joe will be deleted! </b>Then i add postgres user to sudoers (this dockerfile is not meant to be secure, ease of use is what i am looking for when it comes to a lab postgres instance).<br />
I -always- get the latest postgres version sources, compile and install (including extensions , documentation etc) set some environmental variables, configure postgres (postgresql.conf , pg_hba.conf) , put a sample recovery.done there so slaves can get it from pg_basebackup , install some extensions in template1, initdb with checksums, create a replication user and finally copy a very simple replica creation script to the docker image.<br />
<br />
How to make it work<br />
With Docker installed its pretty simple :<br />
<br />
to create dockers own playground network<br />
<i><b>docker network create --subnet=10.0.0.0/16 garden </b> </i><br />
to build the image, from the directory you have the Dockerfile downloaded<br />
<i><b>docker build -t pg_dev . </b></i><br />
to run a container with a specific ip in our kindergarten docker network:<br />
<b>docker run --name master --net garden --ip 10.0.0.2 -i -t pg_dev /bin/bash</b><br />
<b>docker run --name slave1 --net garden --ip 10.0.0.3 -i -t pg_dev /bin/bash</b><br />
<br />
and docker ps should look like this :<br />
<br />
<span style="font-size: x-small;">CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES</span><br />
<span style="font-size: x-small;">7a38d4b11769 pg_dev "/bin/bash" About a minute ago Up About a minute 5432/tcp slave1</span><br />
<span style="font-size: x-small;">2c0266f942ea pg_dev "/bin/bash" About a minute ago Up About a minute 5432/tcp master</span><br />
<br />
Keep in mind that i have started playing around with Docker just 2 days ago. There might be a better way to assign static ips that I'm not aware of (yet).<br />
<br />
Now, say that you have put some data in master and you wanna create a replica on slave1, something like this should work :<br />
<br />
From any slave container home dir :<br />
postgres@7a38d4b11769:~$ ./mk_replica.sh<br />
waiting for server to shut down.... done<br />
server stopped<br />
29044/29044 kB (100%), 1/1 tablespace<br />
server starting<br />
LOG: redirecting log output to logging collector process<br />
HINT: Future log output will appear in directory "pg_log".<br />
<br />
-[ RECORD 1 ]----+--------<br />
lag_bytes | 0<br />
pid | 37<br />
application_name | a_slave<br />
<br />
<br />
The Dockerfile can be found <a href="https://github.com/vventirozos/pgdocker">here</a> and hopefully it can help some people on setting up a lab fast and easy.<br />
<br />
<br />
Thanks for reading.<br />
- Vasilis Ventirozos</div>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com9tag:blogger.com,1999:blog-826910957631479390.post-37380190608326855012016-05-19T02:23:00.000-07:002016-05-19T02:56:27.990-07:00Repairing clog corruptions<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">Yesterday, i got a page from a client about a possible corruption issue to one of his clients, so i decided to share how i dealt with it. Before starting with how i approached the problem, i want to say that corruptions don't just happen in postgres, in <a href="http://www.omniti.com/">OmniTI</a>, we manage A LOT of databases, all with different major versions and in different operating systems and in my time, I haven't seen (many) cases that corruption happened because of a postgres bug (i've seen indexes getting corrupted but i don't remember ever seeing data being corrupted). What i want to say is that corruptions don't just happen, hardware failures, power outages, disks getting full are common reasons for corruptions. </span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">A replica and backups should always be in place and the server should be properly monitored. Unfortunately this server was not managed by us so none of the above was in place..</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">At first I saw in the logs entries like :</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span>
<br />
<div class="">
<b class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">From the logs:</span></b></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">2016-05-18 15:50:06 EDT::@:[10191]: ERROR: could not access status of transaction 128604706</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">2016-05-18 15:50:06 EDT::@:[10191]: DETAIL: Could not open file "pg_clog/007A": No such file or directory.</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">2016-05-18 15:50:06 EDT::@:[10191]: CONTEXT: automatic vacuum of table "pe.pg_toast.pg_toast_4409334"</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">...</span></div>
<div class="">
<b class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">much more to be expected...</span></b></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">...</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">At this point you know that something went wrong, ideally you want to copy your whole $PGDATA to somewhere else and work there while at the same time you start considering uncompressing your most recent backup. In my case, there was no backup and not enough disk space to copy $PGDATA.</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">FUN TIMES :)</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span>
<b><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">I started by fixing all clogs missing which i found logs</span></b><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">dd if=/dev/zero of=/var/db/pgdata/pg_clog/0114 bs=256k count=1</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">dd if=/dev/zero of=/var/db/pgdata/pg_clog/00D1 bs=256k count=1</span><br />
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">dd if=/dev/zero of=/var/db/pgdata/pg_clog/0106 bs=256k count=1</span><br />
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">...</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">...</span></div>
<div class="">
<b class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">keep creating until logs are clean, they can be a lot, in my case they were more than 100....</span></b></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">...</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">...</span></div>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span>
<br />
<div class="">
<b class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">From the logs i also found the tables :</span></b></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">pg_toast.pg_toast_18454</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">pg_toast.pg_toast_35597</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">pg_toast.pg_toast_35607</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">pg_toast.pg_toast_4409334</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">pg_toast.pg_toast_4409344</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">pg_toast.pg_toast_8817516</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span></div>
<div class="">
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">db=# <b class="">select relname,relnamespace from pg_class where oid in (18454,35597,35607,4409334,4409344,8817516) order by relnamespace;</b></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> relname | relnamespace</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">------------------------+--------------</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">table_case1 | 16872</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">table_case1 | 16874</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">table_case2 | 16874</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">table_case2 | 4409063</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">table_case1 | 4409063</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">table_case2 | 8817221</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">(6 rows)</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">db=# <b class="">select oid,nspname from pg_namespace where oid in (16872,16874,16874,4409063,8817221) order by oid;</b></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> oid | nspname</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">---------+------------</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> 16872 | schema1</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> 16874 | schema2</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> 4409063 | schema3</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> 8817221 | schema4</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">(4 rows)</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<b class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">With a join i found schema.table:</span></b></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">schema1.table_case1</span></div>
<div class="">
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">schema2.table_case1</span></div>
<div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">schema2.table_case2</span></div>
</div>
<div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">schema3.table_case2</span></div>
</div>
<div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">schema3.table_case1</span></div>
</div>
<div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">schema4.table_case2</span></div>
</div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span></div>
<div>
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">Basically we have an application with multiple schemas and 2 kinds of tables were corrupted across 4 schemas.</span></div>
</div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span></div>
<div class="">
<div class="">
<b class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">For table_case1 (simple case, table not referenced by anyone):</span></b></div>
<div class="">
<b class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">for each schema :</span></b></div>
<div class="">
<b class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span></b></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">set search_path to schema1;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">create table badrows (badid int);</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">DO $f$</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">declare</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> curid INT := 0;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> vcontent TEXT;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> badid INT;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">begin</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">FOR badid IN SELECT id FROM table_case1 LOOP</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> curid = curid + 1;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> if curid % 100000 = 0 then</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> raise notice '% rows inspected', curid;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> end if;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> begin</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> SELECT id</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> INTO vcontent</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> FROM table_case1 where id = badid;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> exception</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> when others then</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> insert into badrows values (badid);</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span>raise notice 'data for id % is corrupt', badid;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> continue;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> end;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">end loop;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">end;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">$f$;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: x-small;">(This script was taken from </span><span style="background-color: white; color: #222222; font-size: x-small;"><a href="http://www.databasesoup.com/"><b>Josh Berkus </b>blog</a>, and it was modified to fill my needs.)</span></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">create table table_case1_clean as select * from table_case1</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">where id not in (select badid from badrows);</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">TRUNCATE table_case1;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">vacuum full verbose table_case1;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">insert into table_case1 select * from table_case1_clean;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">vacuum full analyze verbose table_case1;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">drop table badrows;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<b class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">For table_case2 (this one is being referenced by 2 other tables) </span></b></div>
<div class="">
<b class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">F or each org (schema):</span></b></div>
<div class="">
<b class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></b></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">set search_path to schema2;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">create table badrows (badid int);</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">DO $f$</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">declare</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> curid INT := 0;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> vcontent TEXT;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> badid INT;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">begin</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">FOR badid IN SELECT id FROM table_case2 LOOP</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> curid = curid + 1;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> if curid % 100000 = 0 then</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> raise notice '% rows inspected', curid;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> end if;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> begin</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> SELECT id</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> INTO vcontent</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> FROM table_case2 where id = badid;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> exception</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> when others then</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> insert into badrows values (badid);</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> raise notice 'data for id % is corrupt', badid;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> continue;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"> end;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">end loop;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">end;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">$f$;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: x-small;">create table table_case2_clean as select * from </span><span style="font-size: x-small;">table_case2</span></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">where id not in (select badid from badrows);</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">alter table reference_table1 drop constraint reference_table1_fkey;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">delete from reference_table1 where table_case2_id in (select badid from badrows) ;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: x-small;">alter table </span><span style="font-size: x-small;">reference_table2</span><span style="font-size: x-small;"> drop constraint reference_table2_fkey;</span></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: x-small;">delete from </span><span style="font-size: x-small;">reference_table2</span><span style="font-size: x-small;"> where </span><span style="font-size: x-small;">table_case2_id</span><span style="font-size: x-small;"> in (select badid from badrows);</span></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: x-small;">TRUNCATE </span><span style="font-size: x-small;">table_case2</span><span style="font-size: x-small;">;</span></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: x-small;">vacuum full verbose </span><span style="font-size: x-small;">table_case2</span><span style="font-size: x-small;">;</span></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: x-small;">insert into </span><span style="font-size: x-small;">table_case2</span><span style="font-size: x-small;"> select * from </span><span style="font-size: x-small;">table_case2_clean</span><span style="font-size: x-small;">;</span></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: x-small;">vacuum full analyze verbose </span><span style="font-size: x-small;">table_case2</span><span style="font-size: x-small;">;</span></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: x-small;">ALTER TABLE ONLY </span><span style="font-size: x-small;">reference_table1</span></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: x-small;"> ADD CONSTRAINT </span><span style="font-size: x-small;">reference_table1_fkey</span><span style="font-size: x-small;"> FOREIGN KEY (</span><span style="font-size: x-small;">table_case2_id</span><span style="font-size: x-small;">) REFERENCES </span><span style="font-size: x-small;">table_case2</span><span style="font-size: x-small;">(id) ON DELETE CASCADE;</span></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: x-small;">ALTER TABLE ONLY </span><span style="font-size: x-small;">reference_table2</span></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><span style="font-size: x-small;"> ADD CONSTRAINT </span><span style="font-size: x-small;">reference_table2_fkey</span><span style="font-size: x-small;"> FOREIGN KEY (</span><span style="font-size: x-small;">table_case2_id</span><span style="font-size: x-small;">) REFERENCES </span><span style="font-size: x-small;">table_case2</span><span style="font-size: x-small;">(id)</span><span style="font-size: x-small;">;</span></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">drop table badrows;</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">(please ignore or report any typos here, i replaced the real table names while i was writing this post so i might messed up with some names).</span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">What we basically did here was to recreate the table without the corrupted rows.</span></div>
<div class="">
<span class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">After this, tables should be corruption free with the minimum possible data loss.</span></span></div>
<div class="">
<span class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">To ensure that you are corruption free you should either pg_dump and restore, or <u class="" style="font-weight: bold;"><i class="">vacuum full</i></u><b> </b>everything<b>, </b><u class="" style="font-weight: bold;">normal vacuum will NOT show corruptions.</u></span></span></div>
<div class="">
<b class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br class="" /></span></b></div>
<div class="">
<span class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"></span></span><br />
<div class="">
<span class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><b>pe=# vacuum verbose schema1.table_case1;</b></span></span></div>
<span class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">
</span></span>
<div class="">
<span class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">INFO: vacuuming " schema1.table_case1"</span></span></div>
<span class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;">
<div class="">
INFO: index " schema1.table_case1_pkey" now contains 12175 row versions in 36 pages</div>
<div class="">
DETAIL: 0 index row versions were removed.</div>
<div class="">
0 index pages have been deleted, 0 are currently reusable.</div>
<div class="">
CPU 0.00s/0.00u sec elapsed 0.15 sec.</div>
<div class="">
INFO: " table_case1": found 0 removable, 12175 nonremovable row versions in 258 out of 258 pages</div>
<div class="">
DETAIL: 0 dead row versions cannot be removed yet.</div>
<div class="">
There were 0 unused item pointers.</div>
<div class="">
0 pages are entirely empty.</div>
<div class="">
CPU 0.00s/0.00u sec elapsed 0.17 sec.</div>
<div class="">
INFO: vacuuming "pg_toast.pg_toast_18454"</div>
<div class="">
INFO: index "pg_toast_18454_index" now contains 51370 row versions in 143 pages</div>
<div class="">
DETAIL: 0 index row versions were removed.</div>
<div class="">
0 index pages have been deleted, 0 are currently reusable.</div>
<div class="">
CPU 0.00s/0.00u sec elapsed 0.00 sec.</div>
<div class="">
INFO: "pg_toast_18454": found 0 removable, 51370 nonremovable row versions in 12331 out of 12331 pages</div>
<div class="">
DETAIL: 0 dead row versions cannot be removed yet.</div>
<div class="">
There were 0 unused item pointers.</div>
<div class="">
0 pages are entirely empty.</div>
<div class="">
CPU 0.05s/0.03u sec elapsed 0.09 sec.</div>
<div class="">
<b>VACUUM</b></div>
<div class="">
pe=#</div>
<div class="">
pe=#</div>
<div class="">
<b>pe=# vacuum full verbose schema1.table_case1;</b></div>
<div class="">
INFO: vacuuming " schema1.table_case1"</div>
<div class="">
<b>ERROR: missing chunk number 0 for toast value 9270408 in pg_toast_18454</b></div>
<div class="">
<b><br /></b></div>
<div class="">
<br /></div>
<div class="">
<br /></div>
<div class="">
Rows were lost, in my case that was acceptable and maybe your case is not the same as mine, but i hope this will provide some guidance in case you get into a similar situation..</div>
</span></span></div>
<div class="">
<b class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span></b></div>
<div class="">
<b class=""><span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span></b></div>
<div class="">
Thanks for reading</div>
<div class="">
- Vasilis</div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif;"><br /></span></div>
</div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span></div>
<div class="">
<span style="font-family: "arial" , "helvetica" , sans-serif; font-size: x-small;"><br /></span></div>
<div class="" style="font-family: Helvetica;">
<span style="font-size: x-small;"><br /></span></div>
</div>
</div>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com0tag:blogger.com,1999:blog-826910957631479390.post-20161431501709761902015-11-24T07:59:00.001-08:002015-11-24T07:59:59.522-08:00Backups Restores and PITR nowdays<div dir="ltr" style="text-align: left;" trbidi="on">
Its been almost a year since 9.4 was released, it included many nice features, but the one that changed my day to day administration was replication slots.<br /><br />In OmniTI , we use something called OmniPITR a lot, OmniPITR could be called a wal management suite, it can be used to wal ship, to replay wals to clean up wals when they are not needed and it can be used for backups. Usually the way I use it is to first set up streaming replication to a replica, and on top of streaming replication setup wal shipping replication, the reason is simple, i want to have all the wals around in case i lose the slave and i didn't really like the wal_keep_segments approach.<br /><br />Backups are usually being taken from a slave in order to reduce load from the master. <br /><br />Briefly, OmniPITR does this by sending a pg_start/stop_backup to the master, puts a pause removal file on the slave so wals are kept and makes 2 tar.gz files for the base backup and the wals.<br /><br />Usually i have a backup server keeping all the wal files and the backups, which means a second (direct or indirect) wal shipping destination.<br />All this is nice and it works really well but with replication slots this could be more simple and more efficient.<br /><br />Recently, I had to setup a backup for a 1 master 2 slave setup on 9.4.<br /><br />The requirements of the backup would be the following:<br /><br /><ul style="text-align: left;">
<li>backup will be taken from the slave.</li>
<li>backup will be incremental allowing PITR</li>
<li>backup will be stored remotely on another machine.</li>
<li>minimum possible wal shipping targets.</li>
<li>replicas should be identical and adding a new replica should be trivial.</li>
<li>backup should be initiated from the server that keeps backups and not from a slave.</li>
<li>backups should be tested by restoration every week.</li>
</ul>
<br /><br />Pretty straight forward stuff really. <br /><br /><br />I setup archiving to the backup server with a simple scp over rsync command,<br /><span style="font-size: x-small;">archive_command = 'rsync -a %p postgres@<Backup server IP>:/data/walarchive/%f'</span><br /><br />I created a backup user that will use .pgpass <br /><span style="font-size: x-small;">touch ~/.pgpass ; chmod 0600 ~/.pgpass</span><br /><br />added :<br /><br /><span style="font-size: x-small;"><master ip>:5432:template1:backup:pgbackup<br /><slave1 ip>:5432:template1:backup:pgbackup<br /><slave2 ip>:5432:template1:backup:pgbackup<br /><slave-x ip>:5432:template1:backup:pgbackup</span><br /><br /><br />Allowed my backup user in pg_hba.conf (in all db servers)<br />and then i simply used pg_basebackup like this :<br /><br /><span style="font-size: x-small;">rm -rf /data/base/* && pg_basebackup -D /data/base/ -h <slave ip> -U backup -Ft -z -U backup && mv /data/base/base.tar.gz /data/backups/basebackup_`date +"%Y-%m-%d"`.tar.gz</span><br />
<br />
<span style="font-size: x-small;"><span style="font-size: small;">I would like if pg_basebackup could customize the backup name and if it used replication slots (coming in 9.5) but none if it is really a problem when it comes to backups.</span></span><br />
<br />
<span style="font-size: x-small;"><span style="font-size: small;">I added a recovery.conf that looks like this :</span></span><br />
<span style="font-size: x-small;"><span style="font-size: small;"> </span> </span><br /><span style="font-size: x-small;">restore_command = '<PATH/TO/pg_standby> -t /data/backups/failover.now -l /data/walarchive %f %p %r'<br />trigger_file = '/data/backups/failover.now'<br />recovery_end_command = 'rm /data/backups/failover.now'<br />recovery_target = 'immediate'</span><br /><br />The parameter “recovery_target” specifies that recovery should end as soon as a consistent state is reached, i.e. as early as possible. When restoring from an online backup, this means the point where taking the backup ended.<br /><br />NOTE that the recovery.conf file will exist if the backup was taken from a slave, always remember to edit it and replace its entries with the ones above.<br /><br />Some notes would be that before you start the database.<br />
Remove all logs from pg_log and verify that:<br />
<br /><b>archive_command ='/bin/true'<br />synchronous_standby_names = ''</b><br />
At this moment all you have to do is to start the database and monitor the log.<br />The log file of a freshly restored database should look like this :<br /><br /><span style="font-size: x-small;">postgres@backup:/data/backups/pg_log$ tail -f postgresql-2015-11-13_020746.log<br />2015-11-13 02:07:46 EET [] [2344]: [1-1] user=,db=,e=00000 LOG: database system was interrupted; last known up at 2015-11-13 02:06:20 EET<br />2015-11-13 02:08:10 EET [] [2344]: [2-1] user=,db=,e=00000 LOG: starting point-in-time recovery to earliest consistent point<br />2015-11-13 02:08:10 EET [] [2344]: [3-1] user=,db=,e=00000 LOG: restored log file "00000002.history" from archive<br />2015-11-13 02:08:10 EET [] [2344]: [4-1] user=,db=,e=00000 LOG: restored log file "000000020000000200000054" from archive<br />2015-11-13 02:08:10 EET [] [2344]: [5-1] user=,db=,e=00000 LOG: redo starts at 2/54000028<br />2015-11-13 02:08:10 EET [] [2344]: [6-1] user=,db=,e=00000 LOG: consistent recovery state reached at 2/540000F0<br />2015-11-13 02:08:10 EET [] [2344]: [7-1] user=,db=,e=00000 LOG: recovery stopping after reaching consistency<br />2015-11-13 02:08:10 EET [] [2344]: [8-1] user=,db=,e=00000 LOG: recovery has paused<br />2015-11-13 02:08:10 EET [] [2344]: [9-1] user=,db=,e=00000 HINT: Execute pg_xlog_replay_resume() to continue.<br />2015-11-13 02:08:10 EET [] [2342]: [3-1] user=,db=,e=00000 LOG: database system is ready to accept read only connections</span><br /><br /><br />at this point run :<br /><br /><span style="font-size: x-small;">psql -c "select pg_xlog_replay_resume()" template1</span><br /><br />you should see in the log file :<br /><br /><span style="font-size: x-small;">2015-11-13 02:10:08 EET [] [2344]: [13-1] user=,db=,e=00000 LOG: archive recovery complete<br />2015-11-13 02:10:09 EET [] [2344]: [14-1] user=,db=,e=00000 LOG: MultiXact member wraparound protections are now enabled<br />2015-11-13 02:10:09 EET [] [2342]: [4-1] user=,db=,e=00000 LOG: database system is ready to accept connections<br />2015-11-13 02:10:09 EET [] [2394]: [1-1] user=,db=,e=00000 LOG: autovacuum launcher started</span><br /><br />Now, assuming that you want to perform PITR.<br /><br />Follow exactly the same restore procedure as previously described but this time the recovery.conf should look like this :<br /><br /><span style="font-size: x-small;">restore_command = '<PATH/TO/pg_standby> -t /data/backups/failover.now -l /data/walarchive %f %p %r'<br /><b>#recovery_target_time = '2015-11-13 00:09:00'</b></span><br />
<span style="font-size: x-small;"><b># or <br />#recovery_target_xid = '1966'</b><br />trigger_file = '/data/backups/failover.now'<br />#recovery_target_inclusive = 'true'<br />recovery_end_command = 'rm /data/backups/failover.now'</span><br /><b>recovery_target_time</b>: This parameter specifies the time stamp up to which recovery will proceed.<br /><br /><b>recovery_target_xid</b>: This parameter specifies the transaction ID up to which recovery will proceed. Keep in mind that while transaction IDs are assigned sequentially at transaction start, transactions can complete in a different numeric order. The transactions that will be recovered are those that committed before (and optionally including) the specified one. The precise stopping point is also influenced by recovery_target_inclusive.<br /><br /><br /><b>recovery_target_inclusive</b>: Specifies whether to stop just after the specified recovery target (true), or just before the recovery target (false). Applies when either recovery_target_time or recovery_target_xid is specified. This setting controls whether transactions having exactly the target commit time or ID, respectively, will be included in the recovery. Default is true.<br />
<br />
<br />
<br />The rest of this procedure should be identical as previously described.<br />
<br />
<br />
<br />
<br />
<br />
<br />Automate the restore procedure on the backup server , set some maintenance crontab entries that will delete old backups and WALs and you are have yourself a very simple but efficient backup strategy.<br />
<br />
Remember , that testing your backups is equally important with backups!<br />
<br />
Thanks for reading<br />
Vasilis</div>
Evol Monkeyhttp://www.blogger.com/profile/01346397854558520528noreply@blogger.com6