Creëren en testen van S3-service

Richt een op MinIO gebaseerde S3-service in een lokale ontwikkelomgeving in en test deze met behulp van Ansible-plays om buckets aan te maken, bestanden te uploaden en te downloaden.

Tags:

Projects: c2platform/phx/ansible , c2platform.core , c2platform.mw

Overzicht

Deze handleiding laat zien hoe je een eenvoudige S3-compatibele service kunt opzetten met MinIO in de PHX ontwikkelomgeving. Het omvat twee Ansible-plays: een om de S3-service op een node genaamd pxd-s3 in te richten, en een andere om deze te testen door een bucket aan te maken, een bestand te uploaden en te downloaden. Dit verifieert de compatibiliteit met de amazon.aws Ansible-collecties voor het beheren van S3-resources.

De opzet draait lokaal met Vagrant en LXD, en simuleert een S3-service die als startpunt kan dienen voor het ontwikkelen van Ansible-rollen, zoals die voor geautomatiseerde binaire downloads in lifecycle management (LCM)-taken.

Het uitvoeren van vagrant up pxd-s3 creëert een LXD-node met Ubuntu 22.04 en rolt een MinIO-container uit. De container exposeert poort 9000 voor de S3-API (compatibel met tools zoals AWS CLI of Ansible’s amazon.aws-modules) en poort 9090 voor de MinIO Console webinterface.

Met de vereisten van de ontwikkelomgeving voldaan, kun je de S3-service in ongeveer 5 minuten inrichten en direct daarna tests uitvoeren.

Randvoorwaarden

Instellen van de PHX Ontwikkelomgeving op Ubuntu 22

Uitrol

Om de S3-node pxd-s3 aan te maken, die een S3-compatibele service biedt op basis van MinIO, voer je de volgende opdracht uit vanuit de root van je lokale kloon van het PHX Ansible-inventarisproject:

vagrant up pxd-s3

Toon mij

Bringing machine 'pxd-s3' up with 'lxd' provider...
==> pxd-s3: Machine has not been created yet, starting...
==> pxd-s3: Importing LXC image...
==> pxd-s3: Mounting shared folders...
    pxd-s3: /vagrant => /home/onknows/git/gitlab/c2/ansible-phx
    pxd-s3: /home/vagrant/.marker => /home/onknows/.marker
    pxd-s3: /home/vagrant/.local/share/marker => /home/onknows/.local/share/marker
    pxd-s3: /root/.marker => /home/onknows/.marker
    pxd-s3: /root/.local/share/marker => /home/onknows/.local/share/marker
    pxd-s3: /software => /software/projects/phx
    pxd-s3: /software-cache => /software/projects/phx/cache
    pxd-s3: /ansible-dev-collections => /home/onknows/git/gitlab/c2/ansible-dev-collections
==> pxd-s3: Waiting for machine to boot. This may take a few minutes...
    pxd-s3: SSH address: 10.190.101.188:22
    pxd-s3: SSH username: vagrant
    pxd-s3: SSH auth method: private key
==> pxd-s3: Machine booted and ready!
==> pxd-s3: Setting hostname...
==> pxd-s3: Running provisioner: shell...
    pxd-s3: Running: inline script
==> pxd-s3: Running provisioner: ansible...
    pxd-s3: Running ansible-playbook...
[DEPRECATION WARNING]: community.general.yaml has been deprecated. The plugin
has been superseded by the the option `result_format=yaml` in callback plugin
ansible.builtin.default from ansible-core 2.13 onwards. This feature will be
removed from community.general in version 13.0.0. Deprecation warnings can be
disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [S3] **********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [pxd-s3]

TASK [Include Linux roles] *****************************************************

TASK [c2platform.core.server_update : include_tasks] ***************************
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/server_update/tasks/update_cache.yml for pxd-s3

TASK [c2platform.core.server_update : Apt update cache] ************************
changed: [pxd-s3]

TASK [c2platform.core.server_update : include_tasks] ***************************
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/server_update/tasks/update.yml for pxd-s3

TASK [c2platform.core.server_update : include_tasks] ***************************
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/server_update/tasks/debian.yml for pxd-s3

TASK [c2platform.core.server_update : Upgrade all packages] ********************
changed: [pxd-s3]

TASK [c2platform.core.server_update : Check reboot] ****************************
ok: [pxd-s3]

TASK [c2platform.core.server_update : Fact server_update_reboot] ***************
ok: [pxd-s3]

TASK [c2platform.core.bootstrap : Include package tasks] ***********************
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/bootstrap/tasks/os.yml for pxd-s3 => (item=['nano', 'wget', 'tree', 'unzip', 'zip', 'jq', 'build-essential', 'python3-dev', 'python3-wheel', 'libsasl2-dev', 'libldap2-dev', 'libssl-dev', 'git', 'git-lfs', 'nfs-common', 'net-tools', 'telnet', 'curl', 'dnsutils', 'python2'])
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/bootstrap/tasks/os.yml for pxd-s3 => (item=python3-pip)
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/bootstrap/tasks/pip.yml for pxd-s3 => (item=['docker', 'requests==2.28.1'])
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/bootstrap/tasks/pip.yml for pxd-s3 => (item=['botocore', 'boto3'])
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/bootstrap/tasks/os.yml for pxd-s3 => (item=['realmd', 'sssd', 'sssd-ad', 'sssd-krb5', 'krb5-user', 'adcli', 'policykit-1', 'sssd-tools', 'libnss-sss', 'libpam-sss', 'bind9-utils', 'samba-common-bin'])

TASK [c2platform.core.bootstrap : OS package] **********************************
changed: [pxd-s3] => (item=['nano', 'wget', 'tree', 'unzip', 'zip', 'jq', 'build-essential', 'python3-dev', 'python3-wheel', 'libsasl2-dev', 'libldap2-dev', 'libssl-dev', 'git', 'git-lfs', 'nfs-common', 'net-tools', 'telnet', 'curl', 'dnsutils', 'python2'])

TASK [c2platform.core.bootstrap : OS package] **********************************
ok: [pxd-s3] => (item=python3-pip)

TASK [c2platform.core.bootstrap : PIP package] *********************************
changed: [pxd-s3] => (item=['docker', 'requests==2.28.1'])

TASK [c2platform.core.bootstrap : PIP package] *********************************
changed: [pxd-s3] => (item=['botocore', 'boto3'])

TASK [c2platform.core.bootstrap : OS package] **********************************
changed: [pxd-s3] => (item=['realmd', 'sssd', 'sssd-ad', 'sssd-krb5', 'krb5-user', 'adcli', 'policykit-1', 'sssd-tools', 'libnss-sss', 'libpam-sss', 'bind9-utils', 'samba-common-bin'])

TASK [c2platform.core.os_trusts : CA distribute ( Debian )] ********************
changed: [pxd-s3] => (item=https://letsencrypt.org/certs/isrgrootx1.pem)
changed: [pxd-s3] => (item=file:///vagrant/.ca/c2/c2.crt)

TASK [c2platform.core.os_trusts : Execute update-ca-certificates ( Debian )] ***
changed: [pxd-s3] => (item=https://letsencrypt.org/certs/isrgrootx1.pem)
changed: [pxd-s3] => (item=file:///vagrant/.ca/c2/c2.crt)

TASK [c2platform.core.secrets : Stat secret dir] *******************************
ok: [pxd-s3 -> localhost] => (item=/home/onknows/git/gitlab/c2/ansible-phx/secret_vars/development)
ok: [pxd-s3 -> localhost] => (item=/runner/project/secret_vars/development)

TASK [c2platform.core.secrets : Include secrets] *******************************
ok: [pxd-s3] => (item=/home/onknows/git/gitlab/c2/ansible-phx/secret_vars/development)

TASK [c2platform.core.linux : Include linux_resources] *************************
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/linux/tasks/fail.yml for pxd-s3 => (item=0_bootstrap Environment pxd-s3 → development)
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/linux/tasks/lineinfile.yml for pxd-s3 => (item=marker Marker)
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/linux/tasks/file.yml for pxd-s3 => (item=marker Python link for marker)

TASK [c2platform.core.linux : Manage lines in text files] **********************
changed: [pxd-s3] => (item=/home/vagrant/.bashrc)
changed: [pxd-s3] => (item=/root/.bashrc)

TASK [c2platform.core.linux : Manage files and file properties] ****************
changed: [pxd-s3] => (item=/usr/bin/python → link)

TASK [geerlingguy.docker : include_tasks] **************************************
included: /home/onknows/git/gitlab/c2/ansible-phx/roles/external/geerlingguy.docker/tasks/setup-Debian.yml for pxd-s3

TASK [geerlingguy.docker : Ensure old versions of Docker are not installed.] ***
ok: [pxd-s3]

TASK [geerlingguy.docker : Ensure dependencies are installed.] *****************
ok: [pxd-s3]

TASK [geerlingguy.docker : Ensure additional dependencies are installed (on Ubuntu >= 20.04).] ***
ok: [pxd-s3]

TASK [geerlingguy.docker : Add Docker apt key.] ********************************
changed: [pxd-s3]

TASK [geerlingguy.docker : Add Docker repository.] *****************************
changed: [pxd-s3]

TASK [geerlingguy.docker : Install Docker (Ansible >=2.12).] *******************
changed: [pxd-s3]

TASK [geerlingguy.docker : Ensure Docker is started and enabled at boot.] ******
ok: [pxd-s3]

RUNNING HANDLER [geerlingguy.docker : restart docker] **************************
changed: [pxd-s3]

TASK [geerlingguy.docker : include_tasks] **************************************
included: /home/onknows/git/gitlab/c2/ansible-phx/roles/external/geerlingguy.docker/tasks/docker-compose.yml for pxd-s3

TASK [geerlingguy.docker : Check current docker-compose version.] **************
ok: [pxd-s3]

TASK [geerlingguy.docker : set_fact] *******************************************
ok: [pxd-s3]

TASK [geerlingguy.docker : Delete existing docker-compose version if it's different.] ***
ok: [pxd-s3]

TASK [geerlingguy.docker : Install Docker Compose (if configured).] ************
changed: [pxd-s3]

TASK [c2platform.mw.docker : Networks] *****************************************
changed: [pxd-s3] => (item=minio)

TASK [c2platform.mw.docker : Images] *******************************************
changed: [pxd-s3] => (item=quay.io/minio/minio)

TASK [c2platform.mw.docker : Volumes] ******************************************
changed: [pxd-s3] => (item=minio)

TASK [c2platform.mw.docker : Container] ****************************************
changed: [pxd-s3] => (item=minio)

TASK [c2platform.core.linux : Include linux_resources] *************************
included: /home/onknows/git/gitlab/c2/ansible-dev-collections/ansible_collections/c2platform/core/roles/linux/tasks/user.yml for pxd-s3 => (item=s3 vagrant)

TASK [c2platform.core.linux : Manage users on a system] ************************
changed: [pxd-s3] => (item=vagrant → present)

PLAY RECAP *********************************************************************
pxd-s3                     : ok=47   changed=20   unreachable=0    failed=0    skipped=33   rescued=0    ignored=0

Deze opdracht gebruikt de Vagrantfile.yml-configuratie om de node in te richten. Het past de playbook plays/mgmt/s3.yml toe, die Docker installeert, de MinIO-image ophaalt en de container start met standaard credentials (admin gebruiker: admin, wachtwoord: Supersecret!).

Na het inrichten kun je de services bereiken op:

S3 API	MinIO Console
`http://192.168.60.14:9000`	`http://192.168.60.14:9090`

S3-testplay

Om de S3-service te testen, voer je de test playbook uit op de pxd-s3-node. Deze play maakt een S3-bucket genaamd ansible aan, downloadt een Tomcat-archief van het internet, uploadt het naar de bucket en downloadt het vervolgens terug om het proces te verifiëren.

Vanuit de root van het Ansible-inventarisproject voer je uit:

PLAY=mgmt/s3_test vagrant provision pxd-s3

Toon mij

==> pxd-s3: Running provisioner: shell...
    pxd-s3: Running: inline script
==> pxd-s3: Running provisioner: ansible...
    pxd-s3: Running ansible-playbook...
[WARNING]: Collection amazon.aws does not support Ansible version 2.16.0
[DEPRECATION WARNING]: community.general.yaml has been deprecated. The plugin 
has been superseded by the the option `result_format=yaml` in callback plugin 
ansible.builtin.default from ansible-core 2.13 onwards. This feature will be 
removed from community.general in version 13.0.0. Deprecation warnings can be 
disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY [S3] **********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [pxd-s3]

TASK [Create "ansible" bucket] *************************************************
changed: [pxd-s3]
[WARNING]: packaging.version Python module not installed, unable to check AWS
SDK versions
[WARNING]: Failed to get bucket public access block settings (not supported by
cloud)
[WARNING]: Failed to get bucket ownership settings (not supported by cloud)
[WARNING]: Failed to get bucket inventory settings (not supported by cloud)

TASK [Download Tomcat from internet] *******************************************
changed: [pxd-s3]

TASK [Upload to S3] ************************************************************
changed: [pxd-s3]

TASK [Download from S3] ********************************************************
changed: [pxd-s3]

PLAY RECAP *********************************************************************
pxd-s3                     : ok=5    changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Dit gebruikt modules uit de amazon.aws Ansible-collecties om te interageren met het S3-endpoint.

Verificatie

Om de testresultaten handmatig te verifiëren, navigeer je naar de MinIO Console webinterface http://192.168.60.14:9090/. Log in met gebruikersnaam admin en wachtwoord Supersecret!.

Na het inloggen zou je de bucket ansible moeten zien met het bestand apache-tomcat-10.1.48.tar.gz.

Review

Deze sectie bekijkt de belangrijkste bestanden in het PHX Ansible-inventarisproject die de S3-opzet en -test mogelijk maken. Ze demonstreren de integratie tussen Vagrant, Ansible en de amazon.aws Ansible-collecties.

Vagrantfile.yml

De Vagrantfile.yml definieert de pxd-s3-node met de ubuntu22-lxd box, wijst IP 192.168.60.14 toe en specificeert de playbook mgmt/s3 voor het inrichten. Vagrant gebruikt dit om de node aan te maken en te configureren.

Vagrantfile.yml

201  - name: s3
202    short_description: S3
203    description: MinIO S3
204    box: ubuntu22-lxd
205    ip-address: 192.168.60.14
206    plays:
207      - mgmt/s3

hosts.ini

Het inventarisbestand hosts.ini definieert Ansible-groepen s3 en s3_download_server. In deze testopzet zit pxd-s3 in beide groepen, en fungeert als zowel de S3-server als de testclient. In productielijke scenario’s zouden deze rollen gescheiden zijn.

hosts.ini

43[s3]
44pxd-s3
45
46[s3_download_server]
47pxd-s3

Playbooks

Twee playbooks worden gebruikt:

plays/mgmt/s3.yml: Richt de MinIO-service in op de s3-groep, en past rollen toe voor Linux-opzet en Docker-beheer.
plays/mgmt/s3_test.yml: Test S3-operaties op de s3-groep met amazon.aws-modules om een bucket aan te maken, te uploaden en een bestand te downloaden.

plays/mgmt/s3.yml

---
- name: S3
  hosts: s3
  become: true

  roles:
    - { role: c2platform.core.linux }
    - { role: geerlingguy.docker, tags: ["docker"] }
    - { role: c2platform.mw.docker, tags: ["docker"] }

plays/mgmt/s3_test.yml

---
- name: S3
  hosts: s3

  vars:
    px_tomcat_download_url: >-
      https://dlcdn.apache.org/tomcat/tomcat-10/v10.1.48/bin/apache-tomcat-10.1.48.tar.gz
    px_upload_src: "/home/vagrant/{{ px_tomcat_download_url | basename }}"
    px_upload_dest: "tomcat/{{ px_tomcat_download_url | basename }}"
    px_download_dest: "/home/vagrant/s3-tomcat.tar.gz"

  tasks:
    - name: Create "ansible" bucket
      amazon.aws.s3_bucket:
        access_key: admin
        secret_key: Supersecret!
        name: ansible
        state: present
        endpoint_url: http://192.168.60.14:9000

    - name: Download Tomcat from internet
      ansible.builtin.get_url:
        url: "{{ px_tomcat_download_url }}"
        dest: "{{ px_upload_src }}"

    - name: Upload to S3
      amazon.aws.s3_object:
        access_key: admin
        secret_key: Supersecret!
        bucket: ansible
        object: "{{ px_upload_dest }}"
        src: "{{ px_upload_src }}"
        mode: put
        endpoint_url: http://192.168.60.14:9000
        encrypt: false

    - name: Download from S3
      amazon.aws.s3_object:
        access_key: admin
        secret_key: Supersecret!
        mode: get
        bucket: ansible
        dest: "{{ px_download_dest }}"
        object: "{{ px_upload_dest }}"
        endpoint_url: http://192.168.60.14:9000

Groepsvariabelen

groepsvariabelen configureren de groepen:

group_vars/s3/main.yml: Definieert Docker-resources voor MinIO, inclusief de image, container en omgevingsvariabelen.
group_vars/s3_download_server/main.yml: Installeert Python-afhankelijkheden zoals botocore en boto3 voor S3-interacties.

group_vars/s3/main.yml

---
bootstrap_packages:
  2_s3:
    - name:
        - docker
        - requests==2.28.1

docker_networks:
  - name: minio

docker_images:
  - name: quay.io/minio/minio
    tag: latest

docker_volumes:
  - name: minio

docker_containers:
  - name: minio
    image: "{{ docker_images[0]['name'] }}:{{ docker_images[0]['tag'] }}"
    ports:
      - "9000:9000"
      - "9090:9090"
    volumes:
      - minio:/data
    restart_policy: unless-stopped
    env:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: Supersecret!  # vault
    command: server /data --console-address ":9090"

linux_resource_groups_disabled: [kerberos]

docker_resources:
  s3:
    - name: vagrant
      module: user
      groups: docker
      append: true

group_vars/s3_download_server/main.yml

---
bootstrap_packages:
  2_s3_download_server:
    - name:
        - botocore
        - boto3

collections/requirements.yml

Dit bestand specificeert vereiste Ansible-collecties, inclusief amazon.aws voor S3-modules.

collections/requirements.yml

13  - name: amazon.aws
14    version: 10.1.2

Aanvullende informatie

Feedback

Was deze pagina nuttig?

Fijn om te horen! Vertel ons alstublieft hoe we kunnen verbeteren.

Jammer om dat te horen. Vertel ons alstublieft hoe we kunnen verbeteren.

Laatst gewijzigd 2025.11.07: phx how-to s3 PHX-270 PHX-275 (5fab55b)