RHCS Dashboard Dev. Env.

Quick Install

1. Clone the Ceph repo (both --depth 1 --single-branch are optional flags but they'll dramatically reduce the amount of data transferred and stored):

   git clone --depth 1 --single-branch git@github.com:ceph/ceph.git

2. Clone this repo:

   git clone https://github.com/rhcs-dashboard/ceph-dev.git

3. Enter ceph-dev directory.
4. To install docker and docker compose, if you're using:
   • Fedora: sudo bash ./docker/scripts/install-docker-compose-fedora.sh
   • CentOS/RHEL: sudo bash ./docker/scripts/install-docker-compose-centos-rhel.sh
   • Other OSes: please check this. Additionally, please ensure that SELinux is running in permissive mode:

     setenforce 0
     sed -i -E 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

5. Use .env.example template for ceph-dev configuration: cp .env.example .env, edit .env and modify CEPH_REPO_DIR=/path/to/... to point to the directory where you cloned the Ceph repo (step #1)
6. Download the container images: docker compose pull
7. Launch it (for a minimal Ceph-only deployment):

   docker compose up -d ceph
   

   or (for Ceph + monitoring stack):

   docker compose up -d
   

8. Check how things are going with docker compose logs -f ceph:
   • After a couple of minutes (aprox.) it'll finally print All done.
9. The dashboard will be available at: https://127.0.0.1:11000 with credentials: admin / admin.

Advanced Installation

• Clone Ceph:

git clone git@github.com:ceph/ceph.git

• Clone rhcs-dashboard/ceph-dev:

git clone git@github.com:rhcs-dashboard/ceph-dev.git

• In ceph-dev, create .env file from template and set your local values:

cd ceph-dev
cp .env.example .env

• Install Docker Compose by running the following, depending on your OS:

# Fedora:
sudo bash ./docker/scripts/install-docker-compose-fedora.sh
# CentOS 7 / RHEL 7:
sudo bash ./docker/scripts/install-docker-compose-centos-rhel.sh

If you ran the above script, then you can run docker and docker compose without sudo if you log out and log in.

• Download docker images:

docker compose pull

• Optionally, set up git pre-commit hook:

docker compose run --rm -e HOST_PWD=$PWD ceph /docker/ci/pre-commit-setup.sh

Usage

You don't need to build ceph if you've set CEPH_IMAGE=rhcsdashboard/ceph-rpm:... container image (the default).

Start ceph + dashboard feature services:

docker compose up -d
# Only ceph:
docker compose up -d ceph

• Display ceph container logs:

docker compose logs -f ceph

• Access dashboard with credentials: admin / admin

https://127.0.0.1:$CEPH_HOST_PORT

Access dev. server dashboard when you see in container logs something like this:

ceph | *********
ceph | All done.
ceph | *********

http://127.0.0.1:$CEPH_PROXY_HOST_PORT

• Restart dashboard module:

docker compose exec ceph /docker/restart-dashboard.sh

• Add OSD in 2nd host (once ceph dashboard is accessible):

docker compose --profile additional-host up -d

• Stop all:

docker compose down

Build Ceph:

# Set a build-ready image and your local ccache path in .env file:
CEPH_IMAGE=rhcsdashboard/ceph:main # DO NOT use ceph-rpm:... image.
HOST_CCACHE_DIR=/path/to/your/local/.ccache
docker compose run --rm ceph /docker/build-ceph.sh

• Rebuild not proxied dashboard frontend:

docker compose run --rm ceph /docker/build-dashboard-frontend.sh

Backend unit tests and/or lint:

# All tests + lint:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_tox
# Tests
# All tests:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_tox py3
# All tests in nautilus branch:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_tox py3-cov,py27-cov
# Only specific tests:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_tox tests/test_rest_client.py tests/test_grafana.py
# Only 1 test:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_tox tests/test_rgw_client.py::RgwClientTest::test_ssl_verify
# Run doctests in 1 file:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_tox run -- pytest --doctest-modules tools.py
# Lint
# All files:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_tox lint
# All files in nautilus branch:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_tox py3-lint,py27-lint
# Only 1 file:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_tox lint controllers/health.py
# Only 1 file in nautilus branch:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_tox py3-run pylint controllers/health.py
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_tox py3-run pycodestyle controllers/health.py
# Other utilities
# Check OpenAPI Specification:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_tox openapi-check
# List tox environmnets:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_tox run tox -lv

• Check dashboard python code with mypy:

# Enable mypy check in .env file:
CHECK_MYPY=1
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_mypy
# Only 1 file:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_mypy src/pybind/mgr/dashboard/controllers/rgw.py

• Run API tests (integration tests based on Teuthology):

# Run tests interactively:
docker compose run --rm -p 11000:11000 ceph bash
source /docker/ci/sanity-checks.sh && create_api_tests_cluster
run_teuthology_tests tasks.mgr.dashboard.test_health
run_teuthology_tests {moreTests}
cleanup_teuthology # this also outputs coverage report.
# All tests:
docker compose run --rm ceph /docker/ci/run-api-tests.sh
# Only specific tests:
docker compose run --rm ceph /docker/ci/run-api-tests.sh tasks.mgr.dashboard.test_health tasks.mgr.dashboard.test_pool
# Only 1 test:
docker compose run --rm ceph /docker/ci/run-api-tests.sh tasks.mgr.dashboard.test_rgw.RgwBucketTest.test_all

Frontend unit tests or lint:

# Tests
# All tests:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_jest
# Only specific tests:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_jest health.component.spec.ts
# Only 1 test:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_jest health.component.spec.ts -t "^HealthComponent should create$"
# Lint
# All files:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_npm_lint

Frontend E2E tests:

# If ceph is running:
docker compose exec ceph /docker/ci/sanity-checks.sh run_frontend_e2e_tests
# Against a running nautilus cluster:
docker compose run --rm -e DASHBOARD_URL=https://ceph:11000 ceph-e2e
# Only 1 specific test file:
docker compose exec ceph /docker/ci/sanity-checks.sh run_frontend_e2e_tests --spec "cypress/integration/ui/dashboard.e2e-spec.ts"
# If ceph is not running:
docker compose run --rm ceph /docker/ci/sanity-checks.sh run_frontend_e2e_tests

Monitoring tests:

docker compose run --rm ceph /docker/ci/sanity-checks.sh run_monitoring

Sanity checks:

docker compose run --rm ceph /docker/ci/run-sanity-checks.sh

Build Ceph documentation:

docker compose run --rm ceph /docker/ci/sanity-checks.sh run_build_doc

Display Ceph documentation:

docker compose run --rm -p 11001:8080 ceph /docker/ci/sanity-checks.sh run_serve_doc
# Access here: http://127.0.0.1:11001

Grafana

If you have started grafana, you can access it at:

https://127.0.0.1:$GRAFANA_HOST_PORT/login

Prometheus

If you have started prometheus, you can access it at:

http://127.0.0.1:$PROMETHEUS_HOST_PORT

Single Sign-On (SSO)

Keycloak (open source Identity and Access Management solution) will be used for authentication, so port 8080 must be available in your machine.

• Start Keycloak:

docker compose --profile sso up -d

You can access Keycloak administration console with credentials: admin / keycloak

http://127.0.0.1:8080 or https://127.0.0.1:8443

• Enable dashboard SSO in running ceph container:

docker compose exec ceph /docker/sso/sso-enable.sh

• Access dashboard with SSO credentials: admin / ssoadmin

https://127.0.0.1:$CEPH_HOST_PORT

• Disable dashboard SSO:

docker compose exec ceph /docker/sso/sso-disable.sh

RGW Multi-Site

• Set appropriate values in .env:

RGW_MULTISITE=1

• Start ceph (cluster 1) + ceph2 (cluster 2):

docker compose --profile ceph2 up -d

• Run 100s duration benchmark:

docker compose exec ceph2 bash
hsbench -a <rgw-user-access-key> -s <rgw-user-secret-key> -u http://127.0.0.1:8000 -z 4K -d 100 -t 10 -b 10

Access local dashboard connected to remote cluster

• Set appropriate values in .env:

DASHBOARD_URL=http://remote.ceph.cluster.com:8443

• Start only ceph:

docker compose up -d ceph

Build and push an image to docker registry:

• Build image:

docker build -t {imageName}:{imageTag} -f {/path/to/Dockerfile} ./docker/ceph
# Examples
# Build base image:
docker build -t rhcsdashboard/ceph-base:centos8 -f ./docker/ceph/centos/Dockerfile ./docker/ceph
# Nightly ceph-rpm main:
docker build -t rhcsdashboard/ceph-rpm:main \
-f ./docker/ceph/rpm/Dockerfile ./docker/ceph \
--network=host
# Using custom repo URL (only for ceph-rpm images):
docker build -t rhcsdashboard/ceph-rpm:v16.2.0 \
-f ./docker/ceph/rpm/Dockerfile ./docker/ceph \
--build-arg REPO_URL=https://download.ceph.com/rpm-pacific/el8/x86_64/ \
--build-arg VCS_BRANCH=v16.2.0 \
--network=host
# Using custom repo files placed in './docker/ceph/rpm' (only for ceph-rpm images):
docker build -t rhcsdashboard/ceph-rpm:rhcs5.0 \
-f ./docker/ceph/rpm/Dockerfile ./docker/ceph \
--build-arg USE_REPO_FILES=1 \
--build-arg VCS_BRANCH=v16.2.0 \
--network=host
# Nautilus E2E image:
docker build -t rhcsdashboard/ceph-e2e:nautilus \
-f ./docker/ceph/e2e/Dockerfile ./docker/ceph \
--build-arg VCS_BRANCH=nautilus \
--network=host

• Optionally, create an additional tag:

docker tag {imageName}:{imageTag} {imageName}:{imageNewTag}
# Example:
docker tag rhcsdashboard/ceph:main rhcsdashboard/ceph:latest

• Log in to rhcs-dashboard docker registry:

docker login -u rhcsdashboard

• Push image:

docker push {imageName}:{imageTag}
# Example:
docker push rhcsdashboard/ceph:main

Start Ceph 2 (useful for parallel development)

• Set your CEPH2_ local values in .env.

• Start ceph2 + ceph + ...:

docker compose --profile ceph2 up -d
# Start ceph2 but not ceph:
docker compose --profile ceph2 up -d --scale ceph=0
# Stop ceph2:
docker compose --profile ceph2 down

Start a downstream ceph product

• Set your DOWNSTREAM_BUILD to a product.

DOWNSTREAM_BUILD=redhat

and start the ceph normally.

docker compose up -d ceph

Kafka + Kafka UI deployment

To start a Kafka and Kafka UI for testing rgw notification specifically, you can start the kafka and kafka-ui by

docker compose up -d kafka

which will start both of them and kafka ui can be visible at http://localhost:8082

HA Proxy

To start ha proxy for mgr service, you can do

docker compose up -d haproxy_mgr

and for rgw service

docker compose up -d haproxy_rgw

By default, rgw is configured to allow CORS.

RGW HA server can be accessed in http://localhost:9080

curl http://localhost:9080
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID></Owner><Buckets></Buckets></ListAllMyBucketsResult>%

Cephalobox

A podman based cephadm deployment for deploying ceph. Instead of a full podman in podman, it leverages podman directly on container's host with privileged capabilities, so that cephadm can natively deploy and manage services.

Quirks

• OSDs are not managed by cephadm. instead deployed using ceph-osd binary.

Deployment

Strongly advised to use podman compose. docker compose is too much..

1. Build cephadm binary in your local. Since this favors mostly local development, generate the cephadm binary in your src/cephadm directory inside ceph.

2. Also build the front-end assets inside the src/pybind/mgr/dashboard/frontend folder in ceph.

3. Populate the CEPH_REPO_DIR in the .env file so that it reads your local code.

4. use the image you want, supports main, umbrella and tentacle. Prebundling the image so that we don't need to pull it everytime we start a cluster. Provide the correct image in .env file's CEPHALOBOX_IMAGE. defaults to rhcsdashboard/cephalobox:main.

5. Start the container

podman compose up -d cephalobox

6. Check the logs. By default, it pushes out cephadm's bootstrap log and also the ceph-mgr logs.

podman compose logs -f cephalobox

7. to destroy

podman compose down -v

Deploying a cluster with custom image

1. Set CEPHADM_IMAGE in the .env file.

CEPHADM_IMAGE=my.registry/ceph/ceph

Optionally, if the registry is behind auth, provide the registry details in its env variables in .env

REGISTRY_URL=
REGISTRY_USERNAME=
REGISTRY_PASSWORD=

You can also set CONTAINER_IMAGE_* for each of the images incase you want to override the defaults

CONTAINER_IMAGE_PROMETHEUS=
CONTAINER_IMAGE_NODE_EXPORTER=
CONTAINER_IMAGE_ALERTMANAGER=
CONTAINER_IMAGE_GRAFANA=

2. Once done all of those, start the cluster as

podman pull cephalobox
podman compose up --attach-dependencies cephalobox

3. The image pulling happens in its own service which can be logged as

podman compose logs -f ceph-image-puller

Disable Shared Ceph Folder

1. Set SHARED_CEPH_REPO_DIR to 0 to disable cephadm from mounting your local directory.

2. Start the cluster

podman compose up --attach-dependencies cephalobox

This will pull the images locally and then mount them to the cephalobox container so that the image pulling is more faster.

Starting multiple clusters

1. Upto 5 clusters are defined in the docker-compose.yml template. This can be started by

podman compose up -d cephalobox cephalobox2 cephalobox3 cephalobox4 cephalobox5

2. You can view individual logs by doing

pdoamn compose logs -f cephalobox4

Or just all the logs by

podman compose logs -f

3. You can also add more clusters there by following the template defined there. Just make sure the port is mapped correctly. For eg. inside docker-compose.yml

cephalobox6:
<<: *cephalobox-basecontainer_name: cephalobox6hostname: cephalobox6ports:
- "8448:8443"
- "3004:3000"

and then start as

podman compose up -d cephalobox6

Limitation

• Impossible to run nvmeof service because of the way it is built.
• Can't create osds at all. Maybe possible to add hosts, but not tested.