Skip to content

feat(BOXP-129): Oracle Cloud Free Tier control plane IaC for lolice cluster - #11581

Open
boxp wants to merge 24 commits into
mainfrom
feature/BOXP-129-oracle-cloud-control-plane
Open

feat(BOXP-129): Oracle Cloud Free Tier control plane IaC for lolice cluster#11581
boxp wants to merge 24 commits into
mainfrom
feature/BOXP-129-oracle-cloud-control-plane

Conversation

@boxp

@boxpboxp commented Jul 23, 2026

Copy link
Copy Markdown
Owner

Summary

lolice cluster の物理 control plane (shanghai-1/2/3) が2台同時障害した場合でも稼働を継続できるよう、Oracle Cloud Free Tier (Ampere A1, 東京, ARM64, ¥0/月) の VM を2台追加するための IaC を実装する。

  • etcd 構成: 物理3 + クラウド2 = 計5台 (クォーラム=3)
  • 物理CP 2台障害時: 残り1物理+2クラウド=3 ≥ クォーラム3 → クラスター継続 ✓
  • コスト: ¥0/月 (Oracle Always Free Tier)

変更内容

新規追加

  • terraform/oci/lolice-control-plane/: OCI Terraform モジュール

    • VCN / Subnet / Internet Gateway / Security List (Tailscale UDP 41641 + SSH 22)
    • VM.Standard.A1.Flex × 2台 (各 2 OCPU / 12 GB RAM, Ubuntu 22.04 ARM64, 東京)
    • cloud-init: Tailscale 自動インストール・認証 (auth key は SSM から取得)
    • S3 バックエンド (tfaction-state), oracle/oci ~> 6.0 プロバイダー
  • ansible/playbooks/cloud-control-plane-join.yml: kubeadm join プレイブック

    • CRI-O + kubelet + kubeadm インストール
    • kubeadm join --control-plane --apiserver-advertise-address=<tailscale-ip> 実行
    • kube-vip は適用しない (L2 VIP は LAN 外無効)
  • docs/project_docs/BOXP-129/plan.md: 設計ドキュメント

既存ファイル更新

  • terraform/tailscale/lolice/acl.tf: tag:cloud-control-plane タグ所有者追加、etcd (2379/2380) / kubelet (10250) / apiserver (6443) の ACL ルール追加
  • terraform/tailscale/lolice/auth_key.tf: クラウド CP 用 Tailscale auth key + SSM パラメータ (/lolice/tailscale/cloud-control-plane-auth-key) 追加
  • ansible/inventories/production/hosts.yml: cloud_control_plane グループ (oracle-cp-1/2) 追加、kube_vip_enabled: false 設定

適用手順

Step 1: Tailscale ACL + auth key 適用

# terraform/tailscale/lolice で apply (auth key が SSM に保存される)cd terraform/tailscale/lolice
terraform apply

Step 2: OCI 認証情報を SSM に保存 (手動)

aws ssm put-parameter --name "/lolice/oci/tenancy-ocid" --value "<TENANCY_OCID>" --type SecureString
aws ssm put-parameter --name "/lolice/oci/user-ocid" --value "<USER_OCID>" --type SecureString
aws ssm put-parameter --name "/lolice/oci/fingerprint" --value "<FINGERPRINT>" --type SecureString
aws ssm put-parameter --name "/lolice/oci/private-key" --value "$(cat ~/.oci/oci_api_key.pem)" --type SecureString

Step 3: Oracle Cloud VM プロビジョニング

cd terraform/oci/lolice-control-plane
terraform apply \
-var="tenancy_ocid=<TENANCY_OCID>" \
-var="user_ocid=<USER_OCID>" \
-var="fingerprint=<FINGERPRINT>" \
-var="private_key=$(cat ~/.oci/oci_api_key.pem)"

Step 4: Ansible inventory の Tailscale IP 更新

terraform output で Tailscale IP を確認し、ansible/inventories/production/hosts.ymlTAILSCALE_IP_PLACEHOLDER_* を更新。

Step 5: kubeadm join

# 既存 CP (shanghai-1) で証明書アップロード
kubeadm init phase upload-certs --upload-certs
kubeadm token create --print-join-command
# Ansible playbook 実行
ansible-playbook playbooks/cloud-control-plane-join.yml \
-i inventories/production \
-e kubeadm_token=<TOKEN> \
-e kubeadm_ca_cert_hash=sha256:<HASH> \
-e kubeadm_certificate_key=<CERT_KEY> \
--limit cloud_control_plane

Test plan

  • terraform plan が terraform/tailscale/lolice と terraform/oci/lolice-control-plane で正常に実行できること
  • Oracle Cloud VM が東京リージョンで起動し、Tailscale に自動登録されること
  • ansible-playbook cloud-control-plane-join.yml で kubeadm join が成功すること
  • etcdctl endpoint health で5メンバーが全て healthy であること
  • shanghai-1/2 停止状態で kubectl get nodes が応答すること (クォーラムテスト)

Closes BOXP-129

🤖 Generated with Claude Code

…luster
Add Terraform and Ansible resources to provision two Oracle Cloud Ampere A1 (ARM64,
Tokyo) nodes as additional Kubernetes control plane members, bringing the etcd cluster
from 3 to 5 members (quorum=3) to survive simultaneous failure of 2 physical CPs.
- terraform/oci/lolice-control-plane/: new OCI Terraform module
- VCN, subnet, internet gateway, security list (Tailscale UDP 41641 + SSH 22)
- VM.Standard.A1.Flex x2 (2 OCPU / 12 GB RAM each, Ubuntu 22.04 ARM64)
- cloud-init: Tailscale auto-registration via auth key from SSM
- S3 backend (tfaction-state), oracle/oci ~> 6.0 provider
- terraform/tailscale/lolice/acl.tf: add tag:cloud-control-plane owner + ACL rules
for etcd (2379/2380), kubelet (10250), and apiserver (6443) between on-prem and cloud
- terraform/tailscale/lolice/auth_key.tf: add reusable/preauthorized auth key for
cloud CP nodes stored in SSM /lolice/tailscale/cloud-control-plane-auth-key
- ansible/inventories/production/hosts.yml: add cloud_control_plane group (oracle-cp-1/2)
with kube_vip_enabled=false (L2 VIP is LAN-only)
- ansible/playbooks/cloud-control-plane-join.yml: kubeadm join playbook for cloud CPs
using Tailscale IP as apiserver-advertise-address
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@boxp-tfaction

This comment has been minimized.

- Use canonical module name community.general.modprobe (fqcn[canonical])
- Add changed_when to kubeadm join command (no-changed-when)
- Convert kubeadm join result tasks to handlers (no-handler)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@boxp-tfaction

This comment has been minimized.

Tailscale tailnet_key description is limited to 50 characters.
Previous value was 66 chars, causing terraform validate to fail.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@github-actions

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

- tfaction-root.yaml: add oracle/oci to available_providers and
terraform/oci/** target group with OCI credential secrets
- acl.tf: restrict reverse rule src from "*" to explicit principals
(autogroup:members + tag:subnet-router + tag:k8s-operator); add
192.168.10.0/24 to autoApprovers.routes so cloud CPs can reach the
LAN VIP 192.168.10.99 via the on-prem subnet router
- outputs.tf: expose Tailscale IP lookup commands for ansible inventory
- cloud-control-plane-join.yml: document subnet router prerequisite
required for cloud CPs to reach cluster_vip 192.168.10.99
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

… plan
- Add registry.terraform.io/oracle/oci to wc-plan.yaml TFPROVIDERCHECK_CONFIG_BODY
(needed after OCI credentials and main branch wc-plan.yaml are updated)
- Set skip_terraform: true in terraform/oci/lolice-control-plane/tfaction.yaml
to bypass terraform init/plan until OCI_TENANCY_OCID, OCI_USER_OCID,
OCI_FINGERPRINT, OCI_PRIVATE_KEY secrets are configured in GitHub
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@boxp-tfaction

This comment has been minimized.

…ion-root.yaml
terraform/oci/** target group needs skip_terraform: true at the root config level
so that list-targets outputs skip_terraform=true in the CI matrix.
Without OCI credentials configured in GitHub secrets, terraform plan would fail.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@boxp-tfaction

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@boxp

boxp commented Aug 14, 2026

Copy link
Copy Markdown
OwnerAuthor

Oracle Cloud credentials 取得手順 (Web Console)

boxpさんからのリクエストに応じて、Oracle Cloud Web Consoleでの詳細な操作手順をまとめます。


必要なcredentials一覧

変数名内容
tenancy_ocidテナンシーのOCID
user_ocidユーザーのOCID
fingerprintAPIキーのフィンガープリント
private_keyPEM形式の秘密鍵 (内容全体)

Step 1: Tenancy OCID を確認する

  1. Oracle Cloud Console にログイン
  2. 右上のプロフィールアイコン → 「テナンシー: <テナンシー名>」 をクリック
  3. 「テナンシー情報」ページが開く
  4. 「OCID」 フィールドの右にある 「コピー」 をクリック
  5. コピーした値が tenancy_ocid (ocid1.tenancy.oc1..xxxx... の形式)

Step 2: ユーザーOCIDを確認する

  1. 右上のプロフィールアイコン → 「プロファイル」 をクリック (または「ユーザー設定」)
  2. 「ユーザーの詳細」ページが開く
  3. 「OCID」 フィールドの右にある 「コピー」 をクリック
  4. コピーした値が user_ocid (ocid1.user.oc1..xxxx... の形式)

Step 3: API キーを作成する (fingerprint と private_key を取得)

  1. 上記「ユーザーの詳細」ページを開いたまま
  2. 左側のメニュー → 「リソース」→「APIキー」 をクリック
  3. 「APIキーの追加」 ボタンをクリック
  4. 「APIキー・ペアの生成」 を選択
  5. 「秘密キーのダウンロード」 ボタンをクリック → oci_api_key.pem ファイルがダウンロードされる
  6. 「追加」 ボタンをクリック
  7. 「構成ファイルのプレビュー」ダイアログが表示される:
    [DEFAULT]user=ocid1.user.oc1..aaaa...
    fingerprint=xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
    tenancy=ocid1.tenancy.oc1..aaaa...
    region=ap-tokyo-1
    key_file=<path to your private keyfile> # TODO
  8. この画面から fingerprint の値 (xx:xx:xx:... の形式) をコピーしておく
  9. 「閉じる」 をクリック

Step 4: GitHub Secrets に登録する (CI/CD用)

GitHub Actions の CI で Terraform plan を実行するために、以下を GitHub Secrets に追加してください:

Secret名
OCI_TENANCY_OCIDStep 1 でコピーした tenancy_ocid
OCI_USER_OCIDStep 2 でコピーした user_ocid
OCI_FINGERPRINTStep 3 でコピーした fingerprint
OCI_PRIVATE_KEYStep 3 でダウンロードした oci_api_key.pemファイル内容全体

OCI_PRIVATE_KEY の登録方法:

# ファイルの内容を確認 (コピーしてGitHub Secretsに貼り付け)
cat ~/Downloads/oci_api_key.pem

-----BEGIN RSA PRIVATE KEY----- から -----END RSA PRIVATE KEY----- まで全て貼り付ける


Step 5: AWS SSM Parameter Store に保存する (Terraform実行用)

Terraform が実際に OCI にリソースを作成するとき用に、AWS SSM にも保存します:

# 以下のコマンドをローカルで実行 (AWS CLI 認証済みの状態で)
aws ssm put-parameter \
--name "/lolice/oci/tenancy-ocid" \
--value "ocid1.tenancy.oc1..ここにStep1の値を貼り付け" \
--type SecureString \
--overwrite
aws ssm put-parameter \
--name "/lolice/oci/user-ocid" \
--value "ocid1.user.oc1..ここにStep2の値を貼り付け" \
--type SecureString \
--overwrite
aws ssm put-parameter \
--name "/lolice/oci/fingerprint" \
--value "xx:xx:xx:ここにStep3のfingerprintを貼り付け" \
--type SecureString \
--overwrite
aws ssm put-parameter \
--name "/lolice/oci/private-key" \
--value "$(cat ~/Downloads/oci_api_key.pem)" \
--type SecureString \
--overwrite

Step 6: Terraform を手動実行する (初回)

CI が通ったら、ローカルで以下を実行してOCI VMを作成します:

cd terraform/oci/lolice-control-plane
# SSMから取得して変数に設定export TF_VAR_tenancy_ocid=$(aws ssm get-parameter --name "/lolice/oci/tenancy-ocid" --with-decryption --query Parameter.Value --output text)export TF_VAR_user_ocid=$(aws ssm get-parameter --name "/lolice/oci/user-ocid" --with-decryption --query Parameter.Value --output text)export TF_VAR_fingerprint=$(aws ssm get-parameter --name "/lolice/oci/fingerprint" --with-decryption --query Parameter.Value --output text)export TF_VAR_private_key=$(aws ssm get-parameter --name "/lolice/oci/private-key" --with-decryption --query Parameter.Value --output text)
terraform init
terraform plan
terraform apply

補足: Compartment OCID について

デフォルトでは compartment_ocid はルートコンパートメント (= tenancy_ocid と同じ値) が使用されます。
別のコンパートメントを使いたい場合:

  1. コンソール左上のハンバーガーメニュー → 「アイデンティティとセキュリティ」「コンパートメント」
  2. 使用したいコンパートメントをクリック → OCIDをコピー
  3. Terraform apply 時に -var="compartment_ocid=ocid1.compartment..." を追加

ご不明点があれば、コメントでお知らせください!

@boxp-tfaction

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@boxp-tfaction

This comment has been minimized.

@github-actions

This comment has been minimized.

@boxp-tfaction

Copy link
Copy Markdown
Contributor

❌ Plan Failed (terraform/oci/lolice-control-plane)

CI link

Error: No value for required variable
on variables.tf line 1:
1: variable "tenancy_ocid" {
The root module input variable "tenancy_ocid" is not set, and has no default
value. Use a -var or -var-file command line argument to provide a value for
this variable.
Error: No value for required variable
on variables.tf line 7:
7: variable "user_ocid" {
The root module input variable "user_ocid" is not set, and has no default
value. Use a -var or -var-file command line argument to provide a value for
this variable.
Error: No value for required variable
on variables.tf line 13:
13: variable "fingerprint" {
The root module input variable "fingerprint" is not set, and has no default
value. Use a -var or -var-file command line argument to provide a value for
this variable.
Error: No value for required variable
on variables.tf line 19:
19: variable "private_key" {
The root module input variable "private_key" is not set, and has no default
value. Use a -var or -var-file command line argument to provide a value for
this variable.

@github-actions

Copy link
Copy Markdown
Contributor

Ansible Plan Results

Mode:--check --diff (dry run)

⚠️Changes detected


golyat-4: worker-image

HostOKChangedSkippedFailedUnreachable
golyat-48615500

1 changed

Changed Tasks (1)
#TaskModule
1user_management : Update package cacheunknown

shanghai-1: control-plane

HostOKChangedSkippedFailedUnreachable
shanghai-110613700

1 changed

Changed Tasks (1)
#TaskModule
1user_management : Update package cacheunknown

shanghai-1: node-shanghai-1

HostOKChangedSkippedFailedUnreachable
shanghai-190000

No changes

shanghai-2: control-plane

HostOKChangedSkippedFailedUnreachable
shanghai-210613700

1 changed

Changed Tasks (1)
#TaskModule
1user_management : Update package cacheunknown

shanghai-2: node-shanghai-2

HostOKChangedSkippedFailedUnreachable
shanghai-290000

No changes

shanghai-3: control-plane

HostOKChangedSkippedFailedUnreachable
shanghai-310613700

1 changed

Changed Tasks (1)
#TaskModule
1user_management : Update package cacheunknown

shanghai-3: node-shanghai-3

HostOKChangedSkippedFailedUnreachable
shanghai-390000

No changes


Plan executed on all nodes in parallel.

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant

@boxp