postgres.md

Latest commit

History

History
255 lines (187 loc) · 8.14 KB

postgres.md

File metadata and controls

255 lines (187 loc) · 8.14 KB
Raw

Using Postgres

Synapse supports PostgreSQL versions 9.6 or later.

Install postgres client libraries

Synapse will require the python postgres client library in order to connect to a postgres database.

  • If you are using the matrix.org debian/ubuntu packages, the necessary python library will already be installed, but you will need to ensure the low-level postgres library is installed, which you can do with apt install libpq5.

  • For other pre-built packages, please consult the documentation from the relevant package.

  • If you installed synapse in a virtualenv, you can install the library with:

    ~/synapse/env/bin/pip install "matrix-synapse[postgres]"

    (substituting the path to your virtualenv for ~/synapse/env, if you used a different path). You will require the postgres development files. These are in the libpq-dev package on Debian-derived distributions.

Set up database

Assuming your PostgreSQL database user is called postgres, first authenticate as the database user with:

su - postgres
# Or, if your system uses sudo to get administrative rights
sudo -u postgres bash

Then, create a postgres user and a database with:

# this will prompt for a password for the new user
createuser --pwprompt synapse_user
createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse

The above will create a user called synapse_user, and a database called synapse.

Note that the PostgreSQL database must have the correct encoding set (as shown above), otherwise it will not be able to store UTF8 strings.

You may need to enable password authentication so synapse_user can connect to the database. See https://www.postgresql.org/docs/current/auth-pg-hba-conf.html.

Synapse config

When you are ready to start using PostgreSQL, edit the database section in your config file to match the following lines:

database:
name: psycopg2args:
user: <user>password: <pass>database: <db>host: <host>cp_min: 5cp_max: 10

All key, values in args are passed to the psycopg2.connect(..) function, except keys beginning with cp_, which are consumed by the twisted adbapi connection pool. See the libpq documentation for a list of options which can be passed.

You should consider tuning the args.keepalives_* options if there is any danger of the connection between your homeserver and database dropping, otherwise Synapse may block for an extended period while it waits for a response from the database server. Example values might be:

database:
args:
# ... as above# seconds of inactivity after which TCP should send a keepalive message to the serverkeepalives_idle: 10# the number of seconds after which a TCP keepalive message that is not# acknowledged by the server should be retransmittedkeepalives_interval: 10# the number of TCP keepalives that can be lost before the client's connection# to the server is considered deadkeepalives_count: 3

Tuning Postgres

The default settings should be fine for most deployments. For larger scale deployments tuning some of the settings is recommended, details of which can be found at https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server.

In particular, we've found tuning the following values helpful for performance:

  • shared_buffers
  • effective_cache_size
  • work_mem
  • maintenance_work_mem
  • autovacuum_work_mem

Note that the appropriate values for those fields depend on the amount of free memory the database host has available.

Porting from SQLite

Overview

The script synapse_port_db allows porting an existing synapse server backed by SQLite to using PostgreSQL. This is done in as a two phase process:

  1. Copy the existing SQLite database to a separate location and run the port script against that offline database.
  2. Shut down the server. Rerun the port script to port any data that has come in since taking the first snapshot. Restart server against the PostgreSQL database.

The port script is designed to be run repeatedly against newer snapshots of the SQLite database file. This makes it safe to repeat step 1 if there was a delay between taking the previous snapshot and being ready to do step 2.

It is safe to at any time kill the port script and restart it.

Note that the database may take up significantly more (25% - 100% more) space on disk after porting to Postgres.

Using the port script

Firstly, shut down the currently running synapse server and copy its database file (typically homeserver.db) to another location. Once the copy is complete, restart synapse. For instance:

./synctl stop
cp homeserver.db homeserver.db.snapshot
./synctl start

Copy the old config file into a new config file:

cp homeserver.yaml homeserver-postgres.yaml

Edit the database section as described in the section Synapse config above and with the SQLite snapshot located at homeserver.db.snapshot simply run:

synapse_port_db --sqlite-database homeserver.db.snapshot \
--postgres-config homeserver-postgres.yaml

The flag --curses displays a coloured curses progress UI.

If the script took a long time to complete, or time has otherwise passed since the original snapshot was taken, repeat the previous steps with a newer snapshot.

To complete the conversion shut down the synapse server and run the port script one last time, e.g. if the SQLite database is at homeserver.db run:

synapse_port_db --sqlite-database homeserver.db \
--postgres-config homeserver-postgres.yaml

Once that has completed, change the synapse config to point at the PostgreSQL database configuration file homeserver-postgres.yaml:

./synctl stop
mv homeserver.yaml homeserver-old-sqlite.yaml
mv homeserver-postgres.yaml homeserver.yaml
./synctl start

Synapse should now be running against PostgreSQL.

Troubleshooting

Alternative auth methods

If you get an error along the lines of FATAL: Ident authentication failed for user "synapse_user", you may need to use an authentication method other than ident:

  • If the synapse_user user has a password, add the password to the database: section of homeserver.yaml. Then add the following to pg_hba.conf:

    host synapse synapse_user ::1/128 md5 # or `scram-sha-256` instead of `md5` if you use that

  • If the synapse_user user does not have a password, then a password doesn't have to be added to homeserver.yaml. But the following does need to be added to pg_hba.conf:

    host synapse synapse_user ::1/128 trust

Note that line order matters in pg_hba.conf, so make sure that if you do add a new line, it is inserted before:

host all all ::1/128 ident

Fixing incorrect COLLATE or CTYPE

Synapse will refuse to set up a new database if it has the wrong values of COLLATE and CTYPE set, and will log warnings on existing databases. Using different locales can cause issues if the locale library is updated from underneath the database, or if a different version of the locale is used on any replicas.

The safest way to fix the issue is to dump the database and recreate it with the correct locale parameter (as shown above). It is also possible to change the parameters on a live database and run a REINDEX on the entire database, however extreme care must be taken to avoid database corruption.

Note that the above may fail with an error about duplicate rows if corruption has already occurred, and such duplicate rows will need to be manually removed.

Fixing inconsistent sequences error

Synapse uses Postgres sequences to generate IDs for various tables. A sequence and associated table can get out of sync if, for example, Synapse has been downgraded and then upgraded again.

To fix the issue shut down Synapse (including any and all workers) and run the SQL command included in the error message. Once done Synapse should start successfully.

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
postgres.md
Latest commit
History
History
255 lines (187 loc) · 8.14 KB
postgres.md
File metadata and controls
255 lines (187 loc) · 8.14 KB
Raw
Using Postgres
Synapse supports PostgreSQL versions 9.6 or later.
Install postgres client libraries
Synapse will require the python postgres client library in order to
connect to a postgres database.
  • If you are using the matrix.org debian/ubuntu
packages, the necessary python
library will already be installed, but you will need to ensure the
low-level postgres library is installed, which you can do with
apt install libpq5.
  • For other pre-built packages, please consult the documentation from
the relevant package.
  • If you installed synapse in a
virtualenv, you can install
the library with:
    ~/synapse/env/bin/pip install "matrix-synapse[postgres]"

    (substituting the path to your virtualenv for ~/synapse/env, if
you used a different path). You will require the postgres
development files. These are in the libpq-dev package on
Debian-derived distributions.
Set up database
Assuming your PostgreSQL database user is called postgres, first authenticate as the database user with:
su - postgres
# Or, if your system uses sudo to get administrative rights
sudo -u postgres bash
Then, create a postgres user and a database with:
# this will prompt for a password for the new user
createuser --pwprompt synapse_user
createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse
The above will create a user called synapse_user, and a database called
synapse.
Note that the PostgreSQL database must have the correct encoding set
(as shown above), otherwise it will not be able to store UTF8 strings.
You may need to enable password authentication so synapse_user can
connect to the database. See
https://www.postgresql.org/docs/current/auth-pg-hba-conf.html.
Synapse config
When you are ready to start using PostgreSQL, edit the database
section in your config file to match the following lines:
database:
name: psycopg2args:
user: <user>password: <pass>database: <db>host: <host>cp_min: 5cp_max: 10
All key, values in args are passed to the psycopg2.connect(..)
function, except keys beginning with cp_, which are consumed by the
twisted adbapi connection pool. See the libpq
documentation
for a list of options which can be passed.
You should consider tuning the args.keepalives_* options if there is any danger of
the connection between your homeserver and database dropping, otherwise Synapse
may block for an extended period while it waits for a response from the
database server. Example values might be:
database:
args:
# ... as above# seconds of inactivity after which TCP should send a keepalive message to the serverkeepalives_idle: 10# the number of seconds after which a TCP keepalive message that is not# acknowledged by the server should be retransmittedkeepalives_interval: 10# the number of TCP keepalives that can be lost before the client's connection# to the server is considered deadkeepalives_count: 3
Tuning Postgres
The default settings should be fine for most deployments. For larger
scale deployments tuning some of the settings is recommended, details of
which can be found at
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server.
In particular, we've found tuning the following values helpful for
performance:
  • shared_buffers
  • effective_cache_size
  • work_mem
  • maintenance_work_mem
  • autovacuum_work_mem
Note that the appropriate values for those fields depend on the amount
of free memory the database host has available.
Porting from SQLite
Overview
The script synapse_port_db allows porting an existing synapse server
backed by SQLite to using PostgreSQL. This is done in as a two phase
process:
  1. Copy the existing SQLite database to a separate location and run
the port script against that offline database.
  2. Shut down the server. Rerun the port script to port any data that
has come in since taking the first snapshot. Restart server against
the PostgreSQL database.
The port script is designed to be run repeatedly against newer snapshots
of the SQLite database file. This makes it safe to repeat step 1 if
there was a delay between taking the previous snapshot and being ready
to do step 2.
It is safe to at any time kill the port script and restart it.
Note that the database may take up significantly more (25% - 100% more)
space on disk after porting to Postgres.
Using the port script
Firstly, shut down the currently running synapse server and copy its
database file (typically homeserver.db) to another location. Once the
copy is complete, restart synapse. For instance:
./synctl stop
cp homeserver.db homeserver.db.snapshot
./synctl start
Copy the old config file into a new config file:
cp homeserver.yaml homeserver-postgres.yaml
Edit the database section as described in the section Synapse config
above and with the SQLite snapshot located at homeserver.db.snapshot
simply run:
synapse_port_db --sqlite-database homeserver.db.snapshot \
--postgres-config homeserver-postgres.yaml
The flag --curses displays a coloured curses progress UI.
If the script took a long time to complete, or time has otherwise passed
since the original snapshot was taken, repeat the previous steps with a
newer snapshot.
To complete the conversion shut down the synapse server and run the port
script one last time, e.g. if the SQLite database is at homeserver.db
run:
synapse_port_db --sqlite-database homeserver.db \
--postgres-config homeserver-postgres.yaml
Once that has completed, change the synapse config to point at the
PostgreSQL database configuration file homeserver-postgres.yaml:
./synctl stop
mv homeserver.yaml homeserver-old-sqlite.yaml
mv homeserver-postgres.yaml homeserver.yaml
./synctl start
Synapse should now be running against PostgreSQL.
Troubleshooting
Alternative auth methods
If you get an error along the lines of FATAL: Ident authentication failed for user "synapse_user", you may need to use an authentication method other than
ident:
  • If the synapse_user user has a password, add the password to the database:
section of homeserver.yaml. Then add the following to pg_hba.conf:
    host synapse synapse_user ::1/128 md5 # or `scram-sha-256` instead of `md5` if you use that

  • If the synapse_user user does not have a password, then a password doesn't
have to be added to homeserver.yaml. But the following does need to be added
to pg_hba.conf:
    host synapse synapse_user ::1/128 trust

Note that line order matters in pg_hba.conf, so make sure that if you do add a
new line, it is inserted before:
host all all ::1/128 ident

Fixing incorrect COLLATE or CTYPE
Synapse will refuse to set up a new database if it has the wrong values of
COLLATE and CTYPE set, and will log warnings on existing databases. Using
different locales can cause issues if the locale library is updated from
underneath the database, or if a different version of the locale is used on any
replicas.
The safest way to fix the issue is to dump the database and recreate it with
the correct locale parameter (as shown above). It is also possible to change the
parameters on a live database and run a REINDEX on the entire database,
however extreme care must be taken to avoid database corruption.
Note that the above may fail with an error about duplicate rows if corruption
has already occurred, and such duplicate rows will need to be manually removed.
Fixing inconsistent sequences error
Synapse uses Postgres sequences to generate IDs for various tables. A sequence
and associated table can get out of sync if, for example, Synapse has been
downgraded and then upgraded again.
To fix the issue shut down Synapse (including any and all workers) and run the
SQL command included in the error message. Once done Synapse should start
successfully.

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
postgres.md
Latest commit
History
History
255 lines (187 loc) · 8.14 KB
postgres.md
File metadata and controls
255 lines (187 loc) · 8.14 KB
Raw
Using Postgres
Synapse supports PostgreSQL versions 9.6 or later.
Install postgres client libraries
Synapse will require the python postgres client library in order to
connect to a postgres database.
  • If you are using the matrix.org debian/ubuntu
packages, the necessary python
library will already be installed, but you will need to ensure the
low-level postgres library is installed, which you can do with
apt install libpq5.
  • For other pre-built packages, please consult the documentation from
the relevant package.
  • If you installed synapse in a
virtualenv, you can install
the library with:
    ~/synapse/env/bin/pip install "matrix-synapse[postgres]"

    (substituting the path to your virtualenv for ~/synapse/env, if
you used a different path). You will require the postgres
development files. These are in the libpq-dev package on
Debian-derived distributions.
Set up database
Assuming your PostgreSQL database user is called postgres, first authenticate as the database user with:
su - postgres
# Or, if your system uses sudo to get administrative rights
sudo -u postgres bash
Then, create a postgres user and a database with:
# this will prompt for a password for the new user
createuser --pwprompt synapse_user
createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse
The above will create a user called synapse_user, and a database called
synapse.
Note that the PostgreSQL database must have the correct encoding set
(as shown above), otherwise it will not be able to store UTF8 strings.
You may need to enable password authentication so synapse_user can
connect to the database. See
https://www.postgresql.org/docs/current/auth-pg-hba-conf.html.
Synapse config
When you are ready to start using PostgreSQL, edit the database
section in your config file to match the following lines:
database:
name: psycopg2args:
user: <user>password: <pass>database: <db>host: <host>cp_min: 5cp_max: 10
All key, values in args are passed to the psycopg2.connect(..)
function, except keys beginning with cp_, which are consumed by the
twisted adbapi connection pool. See the libpq
documentation
for a list of options which can be passed.
You should consider tuning the args.keepalives_* options if there is any danger of
the connection between your homeserver and database dropping, otherwise Synapse
may block for an extended period while it waits for a response from the
database server. Example values might be:
database:
args:
# ... as above# seconds of inactivity after which TCP should send a keepalive message to the serverkeepalives_idle: 10# the number of seconds after which a TCP keepalive message that is not# acknowledged by the server should be retransmittedkeepalives_interval: 10# the number of TCP keepalives that can be lost before the client's connection# to the server is considered deadkeepalives_count: 3
Tuning Postgres
The default settings should be fine for most deployments. For larger
scale deployments tuning some of the settings is recommended, details of
which can be found at
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server.
In particular, we've found tuning the following values helpful for
performance:
  • shared_buffers
  • effective_cache_size
  • work_mem
  • maintenance_work_mem
  • autovacuum_work_mem
Note that the appropriate values for those fields depend on the amount
of free memory the database host has available.
Porting from SQLite
Overview
The script synapse_port_db allows porting an existing synapse server
backed by SQLite to using PostgreSQL. This is done in as a two phase
process:
  1. Copy the existing SQLite database to a separate location and run
the port script against that offline database.
  2. Shut down the server. Rerun the port script to port any data that
has come in since taking the first snapshot. Restart server against
the PostgreSQL database.
The port script is designed to be run repeatedly against newer snapshots
of the SQLite database file. This makes it safe to repeat step 1 if
there was a delay between taking the previous snapshot and being ready
to do step 2.
It is safe to at any time kill the port script and restart it.
Note that the database may take up significantly more (25% - 100% more)
space on disk after porting to Postgres.
Using the port script
Firstly, shut down the currently running synapse server and copy its
database file (typically homeserver.db) to another location. Once the
copy is complete, restart synapse. For instance:
./synctl stop
cp homeserver.db homeserver.db.snapshot
./synctl start
Copy the old config file into a new config file:
cp homeserver.yaml homeserver-postgres.yaml
Edit the database section as described in the section Synapse config
above and with the SQLite snapshot located at homeserver.db.snapshot
simply run:
synapse_port_db --sqlite-database homeserver.db.snapshot \
--postgres-config homeserver-postgres.yaml
The flag --curses displays a coloured curses progress UI.
If the script took a long time to complete, or time has otherwise passed
since the original snapshot was taken, repeat the previous steps with a
newer snapshot.
To complete the conversion shut down the synapse server and run the port
script one last time, e.g. if the SQLite database is at homeserver.db
run:
synapse_port_db --sqlite-database homeserver.db \
--postgres-config homeserver-postgres.yaml
Once that has completed, change the synapse config to point at the
PostgreSQL database configuration file homeserver-postgres.yaml:
./synctl stop
mv homeserver.yaml homeserver-old-sqlite.yaml
mv homeserver-postgres.yaml homeserver.yaml
./synctl start
Synapse should now be running against PostgreSQL.
Troubleshooting
Alternative auth methods
If you get an error along the lines of FATAL: Ident authentication failed for user "synapse_user", you may need to use an authentication method other than
ident:
  • If the synapse_user user has a password, add the password to the database:
section of homeserver.yaml. Then add the following to pg_hba.conf:
    host synapse synapse_user ::1/128 md5 # or `scram-sha-256` instead of `md5` if you use that

  • If the synapse_user user does not have a password, then a password doesn't
have to be added to homeserver.yaml. But the following does need to be added
to pg_hba.conf:
    host synapse synapse_user ::1/128 trust

Note that line order matters in pg_hba.conf, so make sure that if you do add a
new line, it is inserted before:
host all all ::1/128 ident

Fixing incorrect COLLATE or CTYPE
Synapse will refuse to set up a new database if it has the wrong values of
COLLATE and CTYPE set, and will log warnings on existing databases. Using
different locales can cause issues if the locale library is updated from
underneath the database, or if a different version of the locale is used on any
replicas.
The safest way to fix the issue is to dump the database and recreate it with
the correct locale parameter (as shown above). It is also possible to change the
parameters on a live database and run a REINDEX on the entire database,
however extreme care must be taken to avoid database corruption.
Note that the above may fail with an error about duplicate rows if corruption
has already occurred, and such duplicate rows will need to be manually removed.
Fixing inconsistent sequences error
Synapse uses Postgres sequences to generate IDs for various tables. A sequence
and associated table can get out of sync if, for example, Synapse has been
downgraded and then upgraded again.
To fix the issue shut down Synapse (including any and all workers) and run the
SQL command included in the error message. Once done Synapse should start
successfully.

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
postgres.md
Latest commit
History
History
255 lines (187 loc) · 8.14 KB
postgres.md
File metadata and controls
255 lines (187 loc) · 8.14 KB
Raw
Using Postgres
Synapse supports PostgreSQL versions 9.6 or later.
Install postgres client libraries
Synapse will require the python postgres client library in order to
connect to a postgres database.
  • If you are using the matrix.org debian/ubuntu
packages, the necessary python
library will already be installed, but you will need to ensure the
low-level postgres library is installed, which you can do with
apt install libpq5.
  • For other pre-built packages, please consult the documentation from
the relevant package.
  • If you installed synapse in a
virtualenv, you can install
the library with:
    ~/synapse/env/bin/pip install "matrix-synapse[postgres]"

    (substituting the path to your virtualenv for ~/synapse/env, if
you used a different path). You will require the postgres
development files. These are in the libpq-dev package on
Debian-derived distributions.
Set up database
Assuming your PostgreSQL database user is called postgres, first authenticate as the database user with:
su - postgres
# Or, if your system uses sudo to get administrative rights
sudo -u postgres bash
Then, create a postgres user and a database with:
# this will prompt for a password for the new user
createuser --pwprompt synapse_user
createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse
The above will create a user called synapse_user, and a database called
synapse.
Note that the PostgreSQL database must have the correct encoding set
(as shown above), otherwise it will not be able to store UTF8 strings.
You may need to enable password authentication so synapse_user can
connect to the database. See
https://www.postgresql.org/docs/current/auth-pg-hba-conf.html.
Synapse config
When you are ready to start using PostgreSQL, edit the database
section in your config file to match the following lines:
database:
name: psycopg2args:
user: <user>password: <pass>database: <db>host: <host>cp_min: 5cp_max: 10
All key, values in args are passed to the psycopg2.connect(..)
function, except keys beginning with cp_, which are consumed by the
twisted adbapi connection pool. See the libpq
documentation
for a list of options which can be passed.
You should consider tuning the args.keepalives_* options if there is any danger of
the connection between your homeserver and database dropping, otherwise Synapse
may block for an extended period while it waits for a response from the
database server. Example values might be:
database:
args:
# ... as above# seconds of inactivity after which TCP should send a keepalive message to the serverkeepalives_idle: 10# the number of seconds after which a TCP keepalive message that is not# acknowledged by the server should be retransmittedkeepalives_interval: 10# the number of TCP keepalives that can be lost before the client's connection# to the server is considered deadkeepalives_count: 3
Tuning Postgres
The default settings should be fine for most deployments. For larger
scale deployments tuning some of the settings is recommended, details of
which can be found at
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server.
In particular, we've found tuning the following values helpful for
performance:
  • shared_buffers
  • effective_cache_size
  • work_mem
  • maintenance_work_mem
  • autovacuum_work_mem
Note that the appropriate values for those fields depend on the amount
of free memory the database host has available.
Porting from SQLite
Overview
The script synapse_port_db allows porting an existing synapse server
backed by SQLite to using PostgreSQL. This is done in as a two phase
process:
  1. Copy the existing SQLite database to a separate location and run
the port script against that offline database.
  2. Shut down the server. Rerun the port script to port any data that
has come in since taking the first snapshot. Restart server against
the PostgreSQL database.
The port script is designed to be run repeatedly against newer snapshots
of the SQLite database file. This makes it safe to repeat step 1 if
there was a delay between taking the previous snapshot and being ready
to do step 2.
It is safe to at any time kill the port script and restart it.
Note that the database may take up significantly more (25% - 100% more)
space on disk after porting to Postgres.
Using the port script
Firstly, shut down the currently running synapse server and copy its
database file (typically homeserver.db) to another location. Once the
copy is complete, restart synapse. For instance:
./synctl stop
cp homeserver.db homeserver.db.snapshot
./synctl start
Copy the old config file into a new config file:
cp homeserver.yaml homeserver-postgres.yaml
Edit the database section as described in the section Synapse config
above and with the SQLite snapshot located at homeserver.db.snapshot
simply run:
synapse_port_db --sqlite-database homeserver.db.snapshot \
--postgres-config homeserver-postgres.yaml
The flag --curses displays a coloured curses progress UI.
If the script took a long time to complete, or time has otherwise passed
since the original snapshot was taken, repeat the previous steps with a
newer snapshot.
To complete the conversion shut down the synapse server and run the port
script one last time, e.g. if the SQLite database is at homeserver.db
run:
synapse_port_db --sqlite-database homeserver.db \
--postgres-config homeserver-postgres.yaml
Once that has completed, change the synapse config to point at the
PostgreSQL database configuration file homeserver-postgres.yaml:
./synctl stop
mv homeserver.yaml homeserver-old-sqlite.yaml
mv homeserver-postgres.yaml homeserver.yaml
./synctl start
Synapse should now be running against PostgreSQL.
Troubleshooting
Alternative auth methods
If you get an error along the lines of FATAL: Ident authentication failed for user "synapse_user", you may need to use an authentication method other than
ident:
  • If the synapse_user user has a password, add the password to the database:
section of homeserver.yaml. Then add the following to pg_hba.conf:
    host synapse synapse_user ::1/128 md5 # or `scram-sha-256` instead of `md5` if you use that

  • If the synapse_user user does not have a password, then a password doesn't
have to be added to homeserver.yaml. But the following does need to be added
to pg_hba.conf:
    host synapse synapse_user ::1/128 trust

Note that line order matters in pg_hba.conf, so make sure that if you do add a
new line, it is inserted before:
host all all ::1/128 ident

Fixing incorrect COLLATE or CTYPE
Synapse will refuse to set up a new database if it has the wrong values of
COLLATE and CTYPE set, and will log warnings on existing databases. Using
different locales can cause issues if the locale library is updated from
underneath the database, or if a different version of the locale is used on any
replicas.
The safest way to fix the issue is to dump the database and recreate it with
the correct locale parameter (as shown above). It is also possible to change the
parameters on a live database and run a REINDEX on the entire database,
however extreme care must be taken to avoid database corruption.
Note that the above may fail with an error about duplicate rows if corruption
has already occurred, and such duplicate rows will need to be manually removed.
Fixing inconsistent sequences error
Synapse uses Postgres sequences to generate IDs for various tables. A sequence
and associated table can get out of sync if, for example, Synapse has been
downgraded and then upgraded again.
To fix the issue shut down Synapse (including any and all workers) and run the
SQL command included in the error message. Once done Synapse should start
successfully.

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
postgres.md
Latest commit
History
History
255 lines (187 loc) · 8.14 KB
postgres.md
File metadata and controls
255 lines (187 loc) · 8.14 KB
Raw
Using Postgres
Synapse supports PostgreSQL versions 9.6 or later.
Install postgres client libraries
Synapse will require the python postgres client library in order to
connect to a postgres database.
  • If you are using the matrix.org debian/ubuntu
packages, the necessary python
library will already be installed, but you will need to ensure the
low-level postgres library is installed, which you can do with
apt install libpq5.
  • For other pre-built packages, please consult the documentation from
the relevant package.
  • If you installed synapse in a
virtualenv, you can install
the library with:
    ~/synapse/env/bin/pip install "matrix-synapse[postgres]"

    (substituting the path to your virtualenv for ~/synapse/env, if
you used a different path). You will require the postgres
development files. These are in the libpq-dev package on
Debian-derived distributions.
Set up database
Assuming your PostgreSQL database user is called postgres, first authenticate as the database user with:
su - postgres
# Or, if your system uses sudo to get administrative rights
sudo -u postgres bash
Then, create a postgres user and a database with:
# this will prompt for a password for the new user
createuser --pwprompt synapse_user
createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse
The above will create a user called synapse_user, and a database called
synapse.
Note that the PostgreSQL database must have the correct encoding set
(as shown above), otherwise it will not be able to store UTF8 strings.
You may need to enable password authentication so synapse_user can
connect to the database. See
https://www.postgresql.org/docs/current/auth-pg-hba-conf.html.
Synapse config
When you are ready to start using PostgreSQL, edit the database
section in your config file to match the following lines:
database:
name: psycopg2args:
user: <user>password: <pass>database: <db>host: <host>cp_min: 5cp_max: 10
All key, values in args are passed to the psycopg2.connect(..)
function, except keys beginning with cp_, which are consumed by the
twisted adbapi connection pool. See the libpq
documentation
for a list of options which can be passed.
You should consider tuning the args.keepalives_* options if there is any danger of
the connection between your homeserver and database dropping, otherwise Synapse
may block for an extended period while it waits for a response from the
database server. Example values might be:
database:
args:
# ... as above# seconds of inactivity after which TCP should send a keepalive message to the serverkeepalives_idle: 10# the number of seconds after which a TCP keepalive message that is not# acknowledged by the server should be retransmittedkeepalives_interval: 10# the number of TCP keepalives that can be lost before the client's connection# to the server is considered deadkeepalives_count: 3
Tuning Postgres
The default settings should be fine for most deployments. For larger
scale deployments tuning some of the settings is recommended, details of
which can be found at
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server.
In particular, we've found tuning the following values helpful for
performance:
  • shared_buffers
  • effective_cache_size
  • work_mem
  • maintenance_work_mem
  • autovacuum_work_mem
Note that the appropriate values for those fields depend on the amount
of free memory the database host has available.
Porting from SQLite
Overview
The script synapse_port_db allows porting an existing synapse server
backed by SQLite to using PostgreSQL. This is done in as a two phase
process:
  1. Copy the existing SQLite database to a separate location and run
the port script against that offline database.
  2. Shut down the server. Rerun the port script to port any data that
has come in since taking the first snapshot. Restart server against
the PostgreSQL database.
The port script is designed to be run repeatedly against newer snapshots
of the SQLite database file. This makes it safe to repeat step 1 if
there was a delay between taking the previous snapshot and being ready
to do step 2.
It is safe to at any time kill the port script and restart it.
Note that the database may take up significantly more (25% - 100% more)
space on disk after porting to Postgres.
Using the port script
Firstly, shut down the currently running synapse server and copy its
database file (typically homeserver.db) to another location. Once the
copy is complete, restart synapse. For instance:
./synctl stop
cp homeserver.db homeserver.db.snapshot
./synctl start
Copy the old config file into a new config file:
cp homeserver.yaml homeserver-postgres.yaml
Edit the database section as described in the section Synapse config
above and with the SQLite snapshot located at homeserver.db.snapshot
simply run:
synapse_port_db --sqlite-database homeserver.db.snapshot \
--postgres-config homeserver-postgres.yaml
The flag --curses displays a coloured curses progress UI.
If the script took a long time to complete, or time has otherwise passed
since the original snapshot was taken, repeat the previous steps with a
newer snapshot.
To complete the conversion shut down the synapse server and run the port
script one last time, e.g. if the SQLite database is at homeserver.db
run:
synapse_port_db --sqlite-database homeserver.db \
--postgres-config homeserver-postgres.yaml
Once that has completed, change the synapse config to point at the
PostgreSQL database configuration file homeserver-postgres.yaml:
./synctl stop
mv homeserver.yaml homeserver-old-sqlite.yaml
mv homeserver-postgres.yaml homeserver.yaml
./synctl start
Synapse should now be running against PostgreSQL.
Troubleshooting
Alternative auth methods
If you get an error along the lines of FATAL: Ident authentication failed for user "synapse_user", you may need to use an authentication method other than
ident:
  • If the synapse_user user has a password, add the password to the database:
section of homeserver.yaml. Then add the following to pg_hba.conf:
    host synapse synapse_user ::1/128 md5 # or `scram-sha-256` instead of `md5` if you use that

  • If the synapse_user user does not have a password, then a password doesn't
have to be added to homeserver.yaml. But the following does need to be added
to pg_hba.conf:
    host synapse synapse_user ::1/128 trust

Note that line order matters in pg_hba.conf, so make sure that if you do add a
new line, it is inserted before:
host all all ::1/128 ident

Fixing incorrect COLLATE or CTYPE
Synapse will refuse to set up a new database if it has the wrong values of
COLLATE and CTYPE set, and will log warnings on existing databases. Using
different locales can cause issues if the locale library is updated from
underneath the database, or if a different version of the locale is used on any
replicas.
The safest way to fix the issue is to dump the database and recreate it with
the correct locale parameter (as shown above). It is also possible to change the
parameters on a live database and run a REINDEX on the entire database,
however extreme care must be taken to avoid database corruption.
Note that the above may fail with an error about duplicate rows if corruption
has already occurred, and such duplicate rows will need to be manually removed.
Fixing inconsistent sequences error
Synapse uses Postgres sequences to generate IDs for various tables. A sequence
and associated table can get out of sync if, for example, Synapse has been
downgraded and then upgraded again.
To fix the issue shut down Synapse (including any and all workers) and run the
SQL command included in the error message. Once done Synapse should start
successfully.

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
postgres.md
Latest commit
History
History
255 lines (187 loc) · 8.14 KB
postgres.md
File metadata and controls
255 lines (187 loc) · 8.14 KB
Raw
Using Postgres
Synapse supports PostgreSQL versions 9.6 or later.
Install postgres client libraries
Synapse will require the python postgres client library in order to
connect to a postgres database.
  • If you are using the matrix.org debian/ubuntu
packages, the necessary python
library will already be installed, but you will need to ensure the
low-level postgres library is installed, which you can do with
apt install libpq5.
  • For other pre-built packages, please consult the documentation from
the relevant package.
  • If you installed synapse in a
virtualenv, you can install
the library with:
    ~/synapse/env/bin/pip install "matrix-synapse[postgres]"

    (substituting the path to your virtualenv for ~/synapse/env, if
you used a different path). You will require the postgres
development files. These are in the libpq-dev package on
Debian-derived distributions.
Set up database
Assuming your PostgreSQL database user is called postgres, first authenticate as the database user with:
su - postgres
# Or, if your system uses sudo to get administrative rights
sudo -u postgres bash
Then, create a postgres user and a database with:
# this will prompt for a password for the new user
createuser --pwprompt synapse_user
createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse
The above will create a user called synapse_user, and a database called
synapse.
Note that the PostgreSQL database must have the correct encoding set
(as shown above), otherwise it will not be able to store UTF8 strings.
You may need to enable password authentication so synapse_user can
connect to the database. See
https://www.postgresql.org/docs/current/auth-pg-hba-conf.html.
Synapse config
When you are ready to start using PostgreSQL, edit the database
section in your config file to match the following lines:
database:
name: psycopg2args:
user: <user>password: <pass>database: <db>host: <host>cp_min: 5cp_max: 10
All key, values in args are passed to the psycopg2.connect(..)
function, except keys beginning with cp_, which are consumed by the
twisted adbapi connection pool. See the libpq
documentation
for a list of options which can be passed.
You should consider tuning the args.keepalives_* options if there is any danger of
the connection between your homeserver and database dropping, otherwise Synapse
may block for an extended period while it waits for a response from the
database server. Example values might be:
database:
args:
# ... as above# seconds of inactivity after which TCP should send a keepalive message to the serverkeepalives_idle: 10# the number of seconds after which a TCP keepalive message that is not# acknowledged by the server should be retransmittedkeepalives_interval: 10# the number of TCP keepalives that can be lost before the client's connection# to the server is considered deadkeepalives_count: 3
Tuning Postgres
The default settings should be fine for most deployments. For larger
scale deployments tuning some of the settings is recommended, details of
which can be found at
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server.
In particular, we've found tuning the following values helpful for
performance:
  • shared_buffers
  • effective_cache_size
  • work_mem
  • maintenance_work_mem
  • autovacuum_work_mem
Note that the appropriate values for those fields depend on the amount
of free memory the database host has available.
Porting from SQLite
Overview
The script synapse_port_db allows porting an existing synapse server
backed by SQLite to using PostgreSQL. This is done in as a two phase
process:
  1. Copy the existing SQLite database to a separate location and run
the port script against that offline database.
  2. Shut down the server. Rerun the port script to port any data that
has come in since taking the first snapshot. Restart server against
the PostgreSQL database.
The port script is designed to be run repeatedly against newer snapshots
of the SQLite database file. This makes it safe to repeat step 1 if
there was a delay between taking the previous snapshot and being ready
to do step 2.
It is safe to at any time kill the port script and restart it.
Note that the database may take up significantly more (25% - 100% more)
space on disk after porting to Postgres.
Using the port script
Firstly, shut down the currently running synapse server and copy its
database file (typically homeserver.db) to another location. Once the
copy is complete, restart synapse. For instance:
./synctl stop
cp homeserver.db homeserver.db.snapshot
./synctl start
Copy the old config file into a new config file:
cp homeserver.yaml homeserver-postgres.yaml
Edit the database section as described in the section Synapse config
above and with the SQLite snapshot located at homeserver.db.snapshot
simply run:
synapse_port_db --sqlite-database homeserver.db.snapshot \
--postgres-config homeserver-postgres.yaml
The flag --curses displays a coloured curses progress UI.
If the script took a long time to complete, or time has otherwise passed
since the original snapshot was taken, repeat the previous steps with a
newer snapshot.
To complete the conversion shut down the synapse server and run the port
script one last time, e.g. if the SQLite database is at homeserver.db
run:
synapse_port_db --sqlite-database homeserver.db \
--postgres-config homeserver-postgres.yaml
Once that has completed, change the synapse config to point at the
PostgreSQL database configuration file homeserver-postgres.yaml:
./synctl stop
mv homeserver.yaml homeserver-old-sqlite.yaml
mv homeserver-postgres.yaml homeserver.yaml
./synctl start
Synapse should now be running against PostgreSQL.
Troubleshooting
Alternative auth methods
If you get an error along the lines of FATAL: Ident authentication failed for user "synapse_user", you may need to use an authentication method other than
ident:
  • If the synapse_user user has a password, add the password to the database:
section of homeserver.yaml. Then add the following to pg_hba.conf:
    host synapse synapse_user ::1/128 md5 # or `scram-sha-256` instead of `md5` if you use that

  • If the synapse_user user does not have a password, then a password doesn't
have to be added to homeserver.yaml. But the following does need to be added
to pg_hba.conf:
    host synapse synapse_user ::1/128 trust

Note that line order matters in pg_hba.conf, so make sure that if you do add a
new line, it is inserted before:
host all all ::1/128 ident

Fixing incorrect COLLATE or CTYPE
Synapse will refuse to set up a new database if it has the wrong values of
COLLATE and CTYPE set, and will log warnings on existing databases. Using
different locales can cause issues if the locale library is updated from
underneath the database, or if a different version of the locale is used on any
replicas.
The safest way to fix the issue is to dump the database and recreate it with
the correct locale parameter (as shown above). It is also possible to change the
parameters on a live database and run a REINDEX on the entire database,
however extreme care must be taken to avoid database corruption.
Note that the above may fail with an error about duplicate rows if corruption
has already occurred, and such duplicate rows will need to be manually removed.
Fixing inconsistent sequences error
Synapse uses Postgres sequences to generate IDs for various tables. A sequence
and associated table can get out of sync if, for example, Synapse has been
downgraded and then upgraded again.
To fix the issue shut down Synapse (including any and all workers) and run the
SQL command included in the error message. Once done Synapse should start
successfully.

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
postgres.md
Latest commit
History
History
255 lines (187 loc) · 8.14 KB
postgres.md
File metadata and controls
255 lines (187 loc) · 8.14 KB
Raw
Using Postgres
Synapse supports PostgreSQL versions 9.6 or later.
Install postgres client libraries
Synapse will require the python postgres client library in order to
connect to a postgres database.
  • If you are using the matrix.org debian/ubuntu
packages, the necessary python
library will already be installed, but you will need to ensure the
low-level postgres library is installed, which you can do with
apt install libpq5.
  • For other pre-built packages, please consult the documentation from
the relevant package.
  • If you installed synapse in a
virtualenv, you can install
the library with:
    ~/synapse/env/bin/pip install "matrix-synapse[postgres]"

    (substituting the path to your virtualenv for ~/synapse/env, if
you used a different path). You will require the postgres
development files. These are in the libpq-dev package on
Debian-derived distributions.
Set up database
Assuming your PostgreSQL database user is called postgres, first authenticate as the database user with:
su - postgres
# Or, if your system uses sudo to get administrative rights
sudo -u postgres bash
Then, create a postgres user and a database with:
# this will prompt for a password for the new user
createuser --pwprompt synapse_user
createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse
The above will create a user called synapse_user, and a database called
synapse.
Note that the PostgreSQL database must have the correct encoding set
(as shown above), otherwise it will not be able to store UTF8 strings.
You may need to enable password authentication so synapse_user can
connect to the database. See
https://www.postgresql.org/docs/current/auth-pg-hba-conf.html.
Synapse config
When you are ready to start using PostgreSQL, edit the database
section in your config file to match the following lines:
database:
name: psycopg2args:
user: <user>password: <pass>database: <db>host: <host>cp_min: 5cp_max: 10
All key, values in args are passed to the psycopg2.connect(..)
function, except keys beginning with cp_, which are consumed by the
twisted adbapi connection pool. See the libpq
documentation
for a list of options which can be passed.
You should consider tuning the args.keepalives_* options if there is any danger of
the connection between your homeserver and database dropping, otherwise Synapse
may block for an extended period while it waits for a response from the
database server. Example values might be:
database:
args:
# ... as above# seconds of inactivity after which TCP should send a keepalive message to the serverkeepalives_idle: 10# the number of seconds after which a TCP keepalive message that is not# acknowledged by the server should be retransmittedkeepalives_interval: 10# the number of TCP keepalives that can be lost before the client's connection# to the server is considered deadkeepalives_count: 3
Tuning Postgres
The default settings should be fine for most deployments. For larger
scale deployments tuning some of the settings is recommended, details of
which can be found at
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server.
In particular, we've found tuning the following values helpful for
performance:
  • shared_buffers
  • effective_cache_size
  • work_mem
  • maintenance_work_mem
  • autovacuum_work_mem
Note that the appropriate values for those fields depend on the amount
of free memory the database host has available.
Porting from SQLite
Overview
The script synapse_port_db allows porting an existing synapse server
backed by SQLite to using PostgreSQL. This is done in as a two phase
process:
  1. Copy the existing SQLite database to a separate location and run
the port script against that offline database.
  2. Shut down the server. Rerun the port script to port any data that
has come in since taking the first snapshot. Restart server against
the PostgreSQL database.
The port script is designed to be run repeatedly against newer snapshots
of the SQLite database file. This makes it safe to repeat step 1 if
there was a delay between taking the previous snapshot and being ready
to do step 2.
It is safe to at any time kill the port script and restart it.
Note that the database may take up significantly more (25% - 100% more)
space on disk after porting to Postgres.
Using the port script
Firstly, shut down the currently running synapse server and copy its
database file (typically homeserver.db) to another location. Once the
copy is complete, restart synapse. For instance:
./synctl stop
cp homeserver.db homeserver.db.snapshot
./synctl start
Copy the old config file into a new config file:
cp homeserver.yaml homeserver-postgres.yaml
Edit the database section as described in the section Synapse config
above and with the SQLite snapshot located at homeserver.db.snapshot
simply run:
synapse_port_db --sqlite-database homeserver.db.snapshot \
--postgres-config homeserver-postgres.yaml
The flag --curses displays a coloured curses progress UI.
If the script took a long time to complete, or time has otherwise passed
since the original snapshot was taken, repeat the previous steps with a
newer snapshot.
To complete the conversion shut down the synapse server and run the port
script one last time, e.g. if the SQLite database is at homeserver.db
run:
synapse_port_db --sqlite-database homeserver.db \
--postgres-config homeserver-postgres.yaml
Once that has completed, change the synapse config to point at the
PostgreSQL database configuration file homeserver-postgres.yaml:
./synctl stop
mv homeserver.yaml homeserver-old-sqlite.yaml
mv homeserver-postgres.yaml homeserver.yaml
./synctl start
Synapse should now be running against PostgreSQL.
Troubleshooting
Alternative auth methods
If you get an error along the lines of FATAL: Ident authentication failed for user "synapse_user", you may need to use an authentication method other than
ident:
  • If the synapse_user user has a password, add the password to the database:
section of homeserver.yaml. Then add the following to pg_hba.conf:
    host synapse synapse_user ::1/128 md5 # or `scram-sha-256` instead of `md5` if you use that

  • If the synapse_user user does not have a password, then a password doesn't
have to be added to homeserver.yaml. But the following does need to be added
to pg_hba.conf:
    host synapse synapse_user ::1/128 trust

Note that line order matters in pg_hba.conf, so make sure that if you do add a
new line, it is inserted before:
host all all ::1/128 ident

Fixing incorrect COLLATE or CTYPE
Synapse will refuse to set up a new database if it has the wrong values of
COLLATE and CTYPE set, and will log warnings on existing databases. Using
different locales can cause issues if the locale library is updated from
underneath the database, or if a different version of the locale is used on any
replicas.
The safest way to fix the issue is to dump the database and recreate it with
the correct locale parameter (as shown above). It is also possible to change the
parameters on a live database and run a REINDEX on the entire database,
however extreme care must be taken to avoid database corruption.
Note that the above may fail with an error about duplicate rows if corruption
has already occurred, and such duplicate rows will need to be manually removed.
Fixing inconsistent sequences error
Synapse uses Postgres sequences to generate IDs for various tables. A sequence
and associated table can get out of sync if, for example, Synapse has been
downgraded and then upgraded again.
To fix the issue shut down Synapse (including any and all workers) and run the
SQL command included in the error message. Once done Synapse should start
successfully.

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
postgres.md
Latest commit
History
History
255 lines (187 loc) · 8.14 KB
postgres.md
File metadata and controls
255 lines (187 loc) · 8.14 KB
Raw
Using Postgres
Synapse supports PostgreSQL versions 9.6 or later.
Install postgres client libraries
Synapse will require the python postgres client library in order to
connect to a postgres database.
  • If you are using the matrix.org debian/ubuntu
packages, the necessary python
library will already be installed, but you will need to ensure the
low-level postgres library is installed, which you can do with
apt install libpq5.
  • For other pre-built packages, please consult the documentation from
the relevant package.
  • If you installed synapse in a
virtualenv, you can install
the library with:
    ~/synapse/env/bin/pip install "matrix-synapse[postgres]"

    (substituting the path to your virtualenv for ~/synapse/env, if
you used a different path). You will require the postgres
development files. These are in the libpq-dev package on
Debian-derived distributions.
Set up database
Assuming your PostgreSQL database user is called postgres, first authenticate as the database user with:
su - postgres
# Or, if your system uses sudo to get administrative rights
sudo -u postgres bash
Then, create a postgres user and a database with:
# this will prompt for a password for the new user
createuser --pwprompt synapse_user
createdb --encoding=UTF8 --locale=C --template=template0 --owner=synapse_user synapse
The above will create a user called synapse_user, and a database called
synapse.
Note that the PostgreSQL database must have the correct encoding set
(as shown above), otherwise it will not be able to store UTF8 strings.
You may need to enable password authentication so synapse_user can
connect to the database. See
https://www.postgresql.org/docs/current/auth-pg-hba-conf.html.
Synapse config
When you are ready to start using PostgreSQL, edit the database
section in your config file to match the following lines:
database:
name: psycopg2args:
user: <user>password: <pass>database: <db>host: <host>cp_min: 5cp_max: 10
All key, values in args are passed to the psycopg2.connect(..)
function, except keys beginning with cp_, which are consumed by the
twisted adbapi connection pool. See the libpq
documentation
for a list of options which can be passed.
You should consider tuning the args.keepalives_* options if there is any danger of
the connection between your homeserver and database dropping, otherwise Synapse
may block for an extended period while it waits for a response from the
database server. Example values might be:
database:
args:
# ... as above# seconds of inactivity after which TCP should send a keepalive message to the serverkeepalives_idle: 10# the number of seconds after which a TCP keepalive message that is not# acknowledged by the server should be retransmittedkeepalives_interval: 10# the number of TCP keepalives that can be lost before the client's connection# to the server is considered deadkeepalives_count: 3
Tuning Postgres
The default settings should be fine for most deployments. For larger
scale deployments tuning some of the settings is recommended, details of
which can be found at
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server.
In particular, we've found tuning the following values helpful for
performance:
  • shared_buffers
  • effective_cache_size
  • work_mem
  • maintenance_work_mem
  • autovacuum_work_mem
Note that the appropriate values for those fields depend on the amount
of free memory the database host has available.
Porting from SQLite
Overview
The script synapse_port_db allows porting an existing synapse server
backed by SQLite to using PostgreSQL. This is done in as a two phase
process:
  1. Copy the existing SQLite database to a separate location and run
the port script against that offline database.
  2. Shut down the server. Rerun the port script to port any data that
has come in since taking the first snapshot. Restart server against
the PostgreSQL database.
The port script is designed to be run repeatedly against newer snapshots
of the SQLite database file. This makes it safe to repeat step 1 if
there was a delay between taking the previous snapshot and being ready
to do step 2.
It is safe to at any time kill the port script and restart it.
Note that the database may take up significantly more (25% - 100% more)
space on disk after porting to Postgres.
Using the port script
Firstly, shut down the currently running synapse server and copy its
database file (typically homeserver.db) to another location. Once the
copy is complete, restart synapse. For instance:
./synctl stop
cp homeserver.db homeserver.db.snapshot
./synctl start
Copy the old config file into a new config file:
cp homeserver.yaml homeserver-postgres.yaml
Edit the database section as described in the section Synapse config
above and with the SQLite snapshot located at homeserver.db.snapshot
simply run:
synapse_port_db --sqlite-database homeserver.db.snapshot \
--postgres-config homeserver-postgres.yaml
The flag --curses displays a coloured curses progress UI.
If the script took a long time to complete, or time has otherwise passed
since the original snapshot was taken, repeat the previous steps with a
newer snapshot.
To complete the conversion shut down the synapse server and run the port
script one last time, e.g. if the SQLite database is at homeserver.db
run:
synapse_port_db --sqlite-database homeserver.db \
--postgres-config homeserver-postgres.yaml
Once that has completed, change the synapse config to point at the
PostgreSQL database configuration file homeserver-postgres.yaml:
./synctl stop
mv homeserver.yaml homeserver-old-sqlite.yaml
mv homeserver-postgres.yaml homeserver.yaml
./synctl start
Synapse should now be running against PostgreSQL.
Troubleshooting
Alternative auth methods
If you get an error along the lines of FATAL: Ident authentication failed for user "synapse_user", you may need to use an authentication method other than
ident:
  • If the synapse_user user has a password, add the password to the database:
section of homeserver.yaml. Then add the following to pg_hba.conf:
    host synapse synapse_user ::1/128 md5 # or `scram-sha-256` instead of `md5` if you use that

  • If the synapse_user user does not have a password, then a password doesn't
have to be added to homeserver.yaml. But the following does need to be added
to pg_hba.conf:
    host synapse synapse_user ::1/128 trust

Note that line order matters in pg_hba.conf, so make sure that if you do add a
new line, it is inserted before:
host all all ::1/128 ident

Fixing incorrect COLLATE or CTYPE
Synapse will refuse to set up a new database if it has the wrong values of
COLLATE and CTYPE set, and will log warnings on existing databases. Using
different locales can cause issues if the locale library is updated from
underneath the database, or if a different version of the locale is used on any
replicas.
The safest way to fix the issue is to dump the database and recreate it with
the correct locale parameter (as shown above). It is also possible to change the
parameters on a live database and run a REINDEX on the entire database,
however extreme care must be taken to avoid database corruption.
Note that the above may fail with an error about duplicate rows if corruption
has already occurred, and such duplicate rows will need to be manually removed.
Fixing inconsistent sequences error
Synapse uses Postgres sequences to generate IDs for various tables. A sequence
and associated table can get out of sync if, for example, Synapse has been
downgraded and then upgraded again.
To fix the issue shut down Synapse (including any and all workers) and run the
SQL command included in the error message. Once done Synapse should start
successfully.