HOWTO Gitea upgrades, a guide for admins
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

15 KiB

This guide helps Gitea admins perform upgrades safely and provides guidance to troubleshoot problems. It covers Gitea 1.2.0 and above.

There are many ways to install Gitea on various Operating Systems, each with their particular challenges and potential for trouble. The scope of this guide is limited to what Gna! uses daily:

It is the source of truth that Gna! members use when maintaining the infrastructure and developing the service.

Backup

In all cases the only reliable way to perform a backup is with a synchronized point-in-time snapshot of all the storage used by Gitea. For instance, if everything (SQLite database + repositories etc.) is on a single QCOW2 disk attached to a virtual machine, a qemu snapshot guarantees the backup is consistent.

If Gitea uses a S3 storage for attachments with a PostgresQL database, stores Git repositories on a remote file system and queues in a Redis server, the only way to ensure a consistent state is to shutdown Gitea during the backup.

In the simplest case where everything is on a single file system and if the instance is not busy (no mirrors, no users), the backup can be done with:

  • gitea dump to collect everything into one zip file
  • psql/mysql/sqlite dump. Although the zip file created by gitea dump contains a copy of the database it (i) does not work across Gitea versions (it does not have the version table) and (ii) has serious long standing open bugs that may introduce problems when re-injecting the SQL dump in a new database (such as this one or this one or that one or that one).

Verify Gitea works

It is critical to verify that Gitea works very carefully. Restoring the backup done before the upgrade is easy and does not loose any information. But if a problem is discovered days or weeks after the upgrade, it will not be an option and fixing it on a live Gitea instance will be much more challenging.

  • Run gitea doctor --all --log-file /tmp/doctor.log and make sure it does not report any problem. The gitea doctor command is available for Gitea 1.10.5, Gitea 1.10.6 and Gitea >= 1.11.5.
  • Manually verify via the web interface. Making a checklist of a typical use case is a good way to not miss anything.
  • If there is a problem of any kind, increase the log level and take a look at the logs. If you cannot figure out what the problem is, ask for help in the Gna! forum before trying to fix it.

Gitea preparation

  • Manually analyze (reading the patches to the sources of the template directory) and update the customized CSS / content.
  • Do not use gitea help to figure out the location of CustomPath, look at the configuration tab of the administration panel when logged in as an admin.
  • gitea manager flush-queues. If it timesout, run it again with a more generous --timeout argument. It is important because the queues contain serialized data that is not guaranteed to be backward compatible between versions. See this breaking change example to better understand why this is necessary.
  • With Gitea version >=1.17.2
    • go to the web admin panel and pause all queues
  • With Gitea version < 1.17.2
    • Shutdown Gitea so that no new entry is created in the queues

  • Pending question:
    • What happens with non flushable queues?

Docker version

  • Gitea >= 1.17 requires docker >= 20.10.6 otherwise mysterious problems will happen (mysterious in the sense that the problem will manifest itself with a problem different from an error message reading "Docker must be upgraded").

Upgrade and verify

  • If the upgrade is from a gitea version lower than 1.6 and greater or equal to 1.2.0, proceed as follows:
    • Upgrade to 1.2.3 and manually verify it runs
    • Upgrade to 1.4.3 and manually verify it runs
    • Upgrade to 1.5.3 and manually verify it runs
    • Upgrade to 1.6.4 and manually verify it runs
  • If the upgrade is from a gitea version greater or equal to 1.6.4 that is not mentioned to be problematic below, upgrade directly to the latest stable Gitea version, there is no need to upgrade to intermediate versions.
  • Upgrade using the full version number (for instance 1.16.9) and avoid using versions such as "latest" or versions without the minor release number such as 1.17. In theory these shorter version names are associated with a version according to a certain logic: for instance "latest" is supposed to be the latest stable version. But there has been cases in the past where this logic was not implemented correctly and caused unexpected upgrades (1.17.0-rc1 tagged as latest and 1.16.0-rc1 tagged as latest).
  • Verify Gitea works

Troubleshoot

  • Increase the log level. If the Gitea version is recent they can be modified dynamically with gitea manager logging add console -n traceconsole -l TRACE -e '((modules/git)|(services/mirror))'
  • If the upgrade from version x.y to version x.y+2 fails and there is a need to narrow down the problem, try upgrading to the latest minor version of each major version and verify it works. It will show which major version causes the issue and help debug the problem. For instance, if upgrading from 1.14.5 to 1.17.2 does not work:
    • Upgrade from 1.14.5 to 1.14.6 (the last minor version of the 1.14 major version) and verify Gitea works.
    • Upgrade to 1.15.11 (the last minor version of the 1.15 major version) and verify Gitea works.
    • Upgrade to 1.17.2 (the last minor version of the 1.17 major version at this point in time) and verify Gitea works.

List of latest major versions

The database version is stored in the database and can be retrieved with select * from version. If the version found in the database does not match what is in the following table, it probably means the instance was installed from the development tree instead of a package and the database may be in a state that has not been tested for upgrades.

Gitea version Date Database
1.17.2 September 2022 v224
1.16.9 July 2022 v211
1.15.11 January 2022 v189
1.14.7 September 2021 v178
1.13.7 April 2021 v156 Note that the comment in the source incorrectly mention it should be v155.
1.12.6 November 2020 v141 Note that the comment in the source incorrectly mention it should be v140.
1.11.8 June 2020 v118 Note that the comment in the source incorrectly mention it should be v117.
1.10.6 March 2020 v103 Note that the comment in the source incorrectly mention it should be v102.
1.9.6 November 2019 v89 Note that the comment in the source incorrectly mention it should be v88.

When upgrading from a specific version...

  • Any version before 1.17
  • 1.13.0
    • The Webhook shared secret inside the webhook payload has been deprecated and will be removed in 1.14.0: https://github.com/go-gitea/gitea/issues/11755 please use the secret header that uses an hmac signature to validate the webhook payload.
    • Git hooks now default to off! (#13058) In your config, you can check the security section for DISABLE_GIT_HOOKS. To enable them again, you must set the setting to false.

Only when upgrading to a specific version

  • 1.15.2

    • Unfortunately following the release of 1.15.1 it has become apparent that there is an issue with upgrading from 1.15.0 to 1.15.1 due to problem with a table constraint that was unexpectedly automatically dropped. Users who upgraded straight from 1.14.x to 1.15.1 are not affected and users who upgraded from 1.15.0 to 1.15.1 can fix the problem using gitea doctor recreate-table issue_index or upgrade to 1.15.2.
  • Between 1.13.0 1.13.3

    • Password hashing algorithm default has changed back to pbkdf2 from argon2. (#14673)

Unexplained upgrade failures & workarounds

Versions with known issues

Gogs from before September 2015 migrated to Gitea 1.17 may have a dangling pull_repo table and the corresponding pull requests will be removed by gitea doctor --fix --all.

From the 1.11.3 release notes:

  • v1.10.0, v1.10.1, v1.10.2, v1.10.3, v1.10.4, v1.11.0, and v1.11.1 do not use any of these versions, as a bug in the upgrade process will delete attachments from the releases on your repositories.
  • v1.11.2 (now replaced by 1.11.3) was mistakenly compiled with Go 1.14, which Gitea is not currently fully tested with and it’s known to cause a few issues.
  • v1.10.5 (replaced by 1.10.6 if you need to keep using the 1.10 branch) was incorrectly tagged, and was in fact a snapshot of our development branch (1.12-dev). It was also compiled with Go 1.14.

Dump recovery warning

We remind users that a bug was discovered with gitea dump in 1.14.3–1.14.6 and 1.15.0. Database dumps from these versions cause broken fields in the repo_unit and login_source tables causing the issue identified in #16961. Users on 1.14.x must upgrade to 1.14.7 before running gitea dump. If this is not possible and you are affected #17137 provides a new gitea doctor command to fix the repo_unit issue: gitea doctor --fix --run fix-broken-repo-units

Regressions

Regressions explain unexpected behaviors in a range of versions.