AWX Upgrade (postgres 9.6 to 10) 7.0.0 to 9.0.1 fails

ISSUE TYPE

• Bug Report

SUMMARY

The following part in /awx/installer/roles/local_docker/main/tasks/main.yml fails:

- name: Upgrade Postgres
  shell: |
    docker run --rm \
      -v {{ postgres_data_dir }}/pgdata:/var/lib/postgresql/9.6/data \
      -v {{ postgres_data_dir }}/10/data:/var/lib/postgresql/10/data \
      -e PGUSER={{ pg_username }} -e POSTGRES_INITDB_ARGS="-U {{ pg_username }}" \
      tianon/postgres-upgrade:9.6-to-10 --username={{ pg_username }}
  when: upgrade_postgres | bool

ENVIRONMENT

• AWX version: 9.0.1
• AWX install method: docker on linux
• Ansible version: 2.9

STEPS TO REPRODUCE

Upgrading from AWX 7.0.0 to AWX 9.0.1.

Folder permissions (auto generated).

[root@awx2 pgdocker]# ll
total 4
drwxr-xr-x.  3 root    root   18 Nov 26 10:48 10
drwx------. 22 polkitd root 4096 Nov 26 10:53 pgdata

ACTUAL RESULTS

1. First error shown below. I was able fix this adding drive letter to both volume mount points -v {{ postgres_data_dir }}/pgdata:/var/lib/postgresql/9.6/data:Z

["chown: cannot read directory '/var/lib/postgresql/9.6/data': Permission denied", "chown: cannot read directory '/var/lib/postgresql/10/data': Permission denied"]

2. Next error I got shown below.

TASK [local_docker : Upgrade Postgres] *************************************************************************************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "docker run --rm  -v /opt/pgdocker/pgdata:/var/lib/postgresql/9.6/data/upgrade:Z  -v /opt/pgdocker/10/data:/var/lib/postgresql/10/data/upgrade:Z  -e PGUSER=awx -e POSTGRES_INITDB_ARGS=\"-U awx\"  tianon/postgres-upgrade:9.6-to-10 --username=awx\n", "delta": "0:00:26.392647", "end": "2019-11-26 10:49:49.574819", "msg": "non-zero return code", "rc": 1, "start": "2019-11-26 10:49:23.182172", "stderr": "Unable to find image 'tianon/postgres-upgrade:9.6-to-10' locally\nTrying to pull repository docker.io/tianon/postgres-upgrade ... \n9.6-to-10: Pulling from docker.io/tianon/postgres-upgrade\nd599a449871e: Already exists\neadd55e4a4ae: Already exists\n17eea069a47f: Already exists\n22b703021b03: Already exists\n5fa72174baec: Already exists\n338e0b17322b: Already exists\n05fd528a5e36: Already exists\nc224328d751f: Already exists\nf4319784a7ce: Pulling fs layer\n6b0ef024e116: Pulling fs layer\n5bc4176bd4c3: Pulling fs layer\n79c29b3c031a: Pulling fs layer\n63b5329afa6d: Pulling fs layer\n55f0468f4dda: Pulling fs layer\nc4ff909e34cd: Pulling fs layer\n8d9339e0f627: Pulling fs layer\n240ab916af1f: Pulling fs layer\nc6a7751c19c9: Pulling fs layer\n79c29b3c031a: Waiting\n63b5329afa6d: Waiting\n55f0468f4dda: Waiting\nc4ff909e34cd: Waiting\n8d9339e0f627: Waiting\n240ab916af1f: Waiting\nc6a7751c19c9: Waiting\n5bc4176bd4c3: Verifying Checksum\n5bc4176bd4c3: Download complete\n6b0ef024e116: Verifying Checksum\n6b0ef024e116: Download complete\n79c29b3c031a: Verifying Checksum\n79c29b3c031a: Download complete\n63b5329afa6d: Verifying Checksum\n63b5329afa6d: Download complete\n55f0468f4dda: Verifying Checksum\n55f0468f4dda: Download complete\nc4ff909e34cd: Verifying Checksum\nc4ff909e34cd: Download complete\n240ab916af1f: Download complete\nc6a7751c19c9: Verifying Checksum\nc6a7751c19c9: Download complete\n8d9339e0f627: Verifying Checksum\n8d9339e0f627: Download complete\nf4319784a7ce: Verifying Checksum\nf4319784a7ce: Download complete\nf4319784a7ce: Pull complete\n6b0ef024e116: Pull complete\n5bc4176bd4c3: Pull complete\n79c29b3c031a: Pull complete\n63b5329afa6d: Pull complete\n55f0468f4dda: Pull complete\nc4ff909e34cd: Pull complete\n8d9339e0f627: Pull complete\n240ab916af1f: Pull complete\nc6a7751c19c9: Pull complete\nDigest: sha256:2d6b609fcbf7e41e874108dde69345a896b43450875477a74d8548614eba97bd\nStatus: Downloaded newer image for docker.io/tianon/postgres-upgrade:9.6-to-10\ninitdb: directory \"/var/lib/postgresql/10/data\" exists but is not empty\nIf you want to create a new database system, either remove or empty\nthe directory \"/var/lib/postgresql/10/data\" or run initdb\nwith an argument other than \"/var/lib/postgresql/10/data\".", "stderr_lines": ["Unable to find image 'tianon/postgres-upgrade:9.6-to-10' locally", "Trying to pull repository docker.io/tianon/postgres-upgrade ... ", "9.6-to-10: Pulling from docker.io/tianon/postgres-upgrade", "d599a449871e: Already exists", "eadd55e4a4ae: Already exists", "17eea069a47f: Already exists", "22b703021b03: Already exists", "5fa72174baec: Already exists", "338e0b17322b: Already exists", "05fd528a5e36: Already exists", "c224328d751f: Already exists", "f4319784a7ce: Pulling fs layer", "6b0ef024e116: Pulling fs layer", "5bc4176bd4c3: Pulling fs layer", "79c29b3c031a: Pulling fs layer", "63b5329afa6d: Pulling fs layer", "55f0468f4dda: Pulling fs layer", "c4ff909e34cd: Pulling fs layer", "8d9339e0f627: Pulling fs layer", "240ab916af1f: Pulling fs layer", "c6a7751c19c9: Pulling fs layer", "79c29b3c031a: Waiting", "63b5329afa6d: Waiting", "55f0468f4dda: Waiting", "c4ff909e34cd: Waiting", "8d9339e0f627: Waiting", "240ab916af1f: Waiting", "c6a7751c19c9: Waiting", "5bc4176bd4c3: Verifying Checksum", "5bc4176bd4c3: Download complete", "6b0ef024e116: Verifying Checksum", "6b0ef024e116: Download complete", "79c29b3c031a: Verifying Checksum", "79c29b3c031a: Download complete", "63b5329afa6d: Verifying Checksum", "63b5329afa6d: Download complete", "55f0468f4dda: Verifying Checksum", "55f0468f4dda: Download complete", "c4ff909e34cd: Verifying Checksum", "c4ff909e34cd: Download complete", "240ab916af1f: Download complete", "c6a7751c19c9: Verifying Checksum", "c6a7751c19c9: Download complete", "8d9339e0f627: Verifying Checksum", "8d9339e0f627: Download complete", "f4319784a7ce: Verifying Checksum", "f4319784a7ce: Download complete", "f4319784a7ce: Pull complete", "6b0ef024e116: Pull complete", "5bc4176bd4c3: Pull complete", "79c29b3c031a: Pull complete", "63b5329afa6d: Pull complete", "55f0468f4dda: Pull complete", "c4ff909e34cd: Pull complete", "8d9339e0f627: Pull complete", "240ab916af1f: Pull complete", "c6a7751c19c9: Pull complete", "Digest: sha256:2d6b609fcbf7e41e874108dde69345a896b43450875477a74d8548614eba97bd", "Status: Downloaded newer image for docker.io/tianon/postgres-upgrade:9.6-to-10", "initdb: directory \"/var/lib/postgresql/10/data\" exists but is not empty", "If you want to create a new database system, either remove or empty", "the directory \"/var/lib/postgresql/10/data\" or run initdb", "with an argument other than \"/var/lib/postgresql/10/data\"."], "stdout": "The files belonging to this database system will be owned by user \"postgres\".\nThis user must also own the server process.\n\nThe database cluster will be initialized with locale \"en_US.utf8\".\nThe default database encoding has accordingly been set to \"UTF8\".\nThe default text search configuration will be set to \"english\".\n\nData page checksums are disabled.", "stdout_lines": ["The files belonging to this database system will be owned by user \"postgres\".", "This user must also own the server process.", "", "The database cluster will be initialized with locale \"en_US.utf8\".", "The default database encoding has accordingly been set to \"UTF8\".", "The default text search configuration will be set to \"english\".", "", "Data page checksums are disabled."]}

PLAY RECAP *****************************************************************************************************************************************************************************************************************
localhost                  : ok=8    changed=0    unreachable=0    failed=1    skipped=1    rescued=0    ignored=0   

However, running the docker run command manually worked fine and I was able to upgrade succesfully.

docker run --rm -v /opt/pgdocker/pgdata:/var/lib/postgresql/9.6/data:Z -v /opt/pgdocker/pgdatanew:/var/lib/postgresql/10/data:Z -e PGUSER=awx -e POSTGRES_INITDB_ARGS="-U awx" tianon/postgres-upgrade:9.6-to-10 --username=awx

So in order to get the upgrade working I have to skip the ansible playbook upgrade part and run the above docker run command manually.