api: Introducing a new diagnostics API command for system VMs for CloudStack admins by dhlaluku · Pull Request #2721 · apache/cloudstack

dhlaluku · 2018-06-25T15:03:05Z

This feature introduces a new API command that will improve troubleshooting of network issues on CloudStack hosted networks by executing network-utility commands (ping, traceroute, arping) remotely on system VMs.

Description

For troubleshooting purposes, CloudStack administrators may wish to execute network utility commands remotely on system VMs, or request system VMs to ping/traceroute/arping to specific addresses over specific interfaces. An API command to provide such functionalities is being developed without altering any existing APIs. The targeted system VMs for this feature are the Virtual Router (VR), Secondary Storage VM (SSVM) and the Console Proxy VM (CPVM).

FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Remote+Diagnostics+API
ML discussion: https://markmail.org/message/xt7owmb2c6iw7tva

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

GitHub Issue/PRs

Screenshots (if appropriate):

How Has This Been Tested?

Log on to the CloudStack management server as root admin and start cloudmonkey. Sync APIs and then execute the run diagnostics command as follows:
run diganostics ipaddress=www.shapeblue.com type=arping targetid=uuid params="-I eth0 -c 4"

Where;

ipaddress is the destination IP/Domain address to test connection to;
type is the diagnostics command type to execute from remote target
targetid is the uuid of the system VM from which to test
params are optional command line arguments that apply to each diagnostics command type

Dev environment components:

Platform: ACS-4.12
Management OS: Ubuntu 18.04 LTS
Hypervisor: 1 Host running KVM with CentOS 7

This command does not run with the CloudStack simulator and has only been tested on real hardware environments.

Checklist:

I have read the CONTRIBUTING document.
My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
Testing
I have added tests to cover my changes.
All relevant new and existing integration tests have passed.
A full integration testsuite with all test that can run on my environment has passed.

borisstoyanov · 2018-06-26T06:03:51Z

@blueorangutan package

blueorangutan · 2018-06-26T06:04:43Z

@borisstoyanov a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

borisstoyanov

I've executed the following manual tests:

Test Name	Steps
Run against CPVM	With cloudmonkey execute: run diagnostics type=arping ipaddress=www.shapeblue.com id=[CPVM-id] params="-I eth1 -c 8"
Run against SSVM	With cloudmonkey execute: run diagnostics type=arping ipaddress=www.shapeblue.com id=[SSVM-id] params="-I eth1 -c 8"
Run against VR	With cloudmonkey execute: run diagnostics type=arping ipaddress=www.shapeblue.com id=[vr-id] params="-I eth1 -c 8"
Run a ping type	With cloudmonkey execute: run diagnostics type=ping ipaddress=www.shapeblue.com id=[vr-id]
Run an arping type	With cloudmonkey execute: run diagnostics type=arping ipaddress=www.shapeblue.com id=[vr-id] params="-I eth1 -c 8"
Run a traceroute type	With cloudmonkey execute: run diagnostics type=traceroute ipaddress=www.shapeblue.com id=[vr-id]
Run against VR with arguments	With cloudmonkey execute: run diagnostics type=arping ipaddress=www.shapeblue.com id=[vr-id] params="-I eth1 -c 8"
Run a ping type with arguments	With cloudmonkey execute: run diagnostics type=arping ipaddress=www.shapeblue.com id=[vr-id] params="-I eth1 -c 8"
Run an traceroute type with arguments	With cloudmonkey execute: run diagnostics type=arping ipaddress=www.shapeblue.com id=[vr-id] params="-I eth1 -c 8"
Run an type with invalid arguments	With cloudmonkey execute: run diagnostics type=arping ipaddress=www.shapeblue.com id=[vr-id] params="-H eth1 -c 8"
Run type to unreachable desctination	With cloudmonkey execute: run diagnostics type=arping ipaddress=www.something-not-readchable.com id=[vr-id] params="-H eth1 -c 8"
Run with invalid type	run diagnostics type=netstat ipaddress=www.shapeblue.com id=[vr-id]
Run with invalid target id.	run diagnostics type=ping ipaddress=www.shapeblue.com id=[invalid-id]

LGTM, lets wait for the automated results

blueorangutan · 2018-06-26T06:36:12Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2152

borisstoyanov · 2018-06-26T06:39:26Z

@blueorangutan test

blueorangutan · 2018-06-26T06:39:44Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2018-06-26T14:33:17Z

Trillian test result (tid-2825)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 26692 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2721-t2825-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_accounts.py
Intermitten failure detected: /marvin/tests/smoke/test_affinity_groups_projects.py
Intermitten failure detected: /marvin/tests/smoke/test_certauthority_root.py
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_primary_storage.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_snapshots.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_host_maintenance.py
Intermitten failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 58 look OK, 10 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestAccounts>:setup	`Error`	0.00	test_accounts.py
ContextSuite context=TestAddVmToSubDomain>:setup	`Error`	0.00	test_accounts.py
test_DeleteDomain	`Error`	0.92	test_accounts.py
test_forceDeleteDomain	`Error`	0.94	test_accounts.py
ContextSuite context=TestRemoveUserFromAccount>:setup	`Error`	5.14	test_accounts.py
ContextSuite context=TestDeployVmWithAffinityGroup>:setup	`Error`	0.00	test_affinity_groups_projects.py
test_provision_certificate	`Error`	17.38	test_certauthority_root.py
ContextSuite context=TestDeployVirtioSCSIVM>:setup	`Error`	0.00	test_deploy_virtio_scsi_vm.py
test_01_add_primary_storage_disabled_host	`Error`	0.73	test_primary_storage.py
test_01_primary_storage_nfs	`Error`	0.10	test_primary_storage.py
ContextSuite context=TestStorageTags>:setup	`Error`	0.18	test_primary_storage.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	148.40	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	249.85	test_privategw_acl.py
test_02_list_snapshots_with_removed_data_store	`Error`	1.13	test_snapshots.py
test_01_secure_vm_migration	`Error`	5.16	test_vm_life_cycle.py
test_02_unsecure_vm_migration	`Error`	5.62	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	1.14	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	4.59	test_vm_life_cycle.py
ContextSuite context=TestVMLifeCycle>:setup	`Error`	7.35	test_vm_life_cycle.py
test_01_cancel_host_maintenace_with_no_migration_jobs	`Failure`	0.09	test_host_maintenance.py
test_02_cancel_host_maintenace_with_migration_jobs	`Error`	2.25	test_host_maintenance.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	2.46	test_hostha_kvm.py

borisstoyanov · 2018-06-27T07:13:28Z

@blueorangutan test

blueorangutan · 2018-06-27T07:13:44Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

borisstoyanov · 2018-06-28T09:57:19Z

@blueorangutan test

blueorangutan · 2018-06-28T09:57:44Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2018-06-28T18:35:05Z

Trillian test result (tid-2830)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 29372 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2721-t2830-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_certauthority_root.py
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_volumes.py
Intermitten failure detected: /marvin/tests/smoke/test_host_maintenance.py
Intermitten failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 62 look OK, 6 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_provision_certificate	`Error`	8.21	test_certauthority_root.py
ContextSuite context=TestDeployVirtioSCSIVM>:setup	`Error`	0.00	test_deploy_virtio_scsi_vm.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	1213.03	test_privategw_acl.py
test_01_secure_vm_migration	`Error`	5.16	test_vm_life_cycle.py
test_02_unsecure_vm_migration	`Error`	1.08	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	1.08	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	3.12	test_vm_life_cycle.py
test_11_migrate_volume_and_change_offering	`Error`	129.66	test_volumes.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	3.46	test_hostha_kvm.py
test_hostha_kvm_host_recovering	`Error`	7.56	test_hostha_kvm.py

borisstoyanov · 2018-07-02T07:46:09Z

@blueorangutan test

blueorangutan · 2018-07-02T07:46:46Z

@borisstoyanov a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2018-07-02T16:14:28Z

Trillian test result (tid-2831)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 28543 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2721-t2831-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_certauthority_root.py
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_volumes.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 62 look OK, 6 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_provision_certificate	`Error`	7.25	test_certauthority_root.py
ContextSuite context=TestDeployVirtioSCSIVM>:setup	`Error`	0.00	test_deploy_virtio_scsi_vm.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	1157.31	test_privategw_acl.py
test_01_secure_vm_migration	`Error`	3.15	test_vm_life_cycle.py
test_02_unsecure_vm_migration	`Error`	1.10	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	1.10	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	3.19	test_vm_life_cycle.py
test_11_migrate_volume_and_change_offering	`Error`	128.27	test_volumes.py
test_02_redundant_VPC_default_routes	`Failure`	990.47	test_vpc_redundant.py

DaanHoogland

in the niche of this new feature we might find aspects we would like slightly different. The code looks good and contains unit - and integration tests. Let's run with it.

yadvr · 2018-07-04T10:20:43Z

+from nose.plugins.attrib import attr
+
+
+class TestRemoteDiagnostics(cloudstackTestCase):


@dhlaluku can you add this to .travis.yml, fix it to make it work/skip against simulator?

Tests has been added to Travis file in the section that contains tests for routers and system VMs

yadvr

@dhlaluku for simulator see if you add some mocks to get this APIs work against simulator i.e. with Travis to test business logic. Please see if you can fix that.

yadvr · 2018-07-06T08:12:59Z

Please hold merging, while everything is LGTM -- @dhlaluku will add support for simulator with his work to make the business logic test-able and add his new marvin test with .travis.yml

troubleshooting of network issues in CloudStack hosted networks

yadvr · 2018-07-10T06:22:47Z

ping @dhlaluku

dhlaluku · 2018-07-10T10:46:34Z

@rhtyd I have included "test_diagnostics" in the Travis file. Updated the Marvin tests to skip the negative test cases for the Simulator hypervisor.

Also defined some mocks for the positive cases, example below with Cloudmonkey;

(local) 🐵 > run diagnostics targetid=0c04cf04-8222-47fd-a883-6cf335646d64 ipaddress=8.8.8.8 type=traceroute
{
"diagnostics": {
"exitcode": "0",
"stderr": "",
"stdout": "TRACEROUTE 8.8.8.8 executed in v-1-VM"
}
}

yadvr · 2018-07-10T10:47:58Z

Fantastic @dhlaluku - let's wait for travis to complete/pass.

yadvr · 2018-07-10T10:48:45Z

 # Keep the TESTS sorted by name and grouped by type
 - TESTS="smoke/test_certauthority_root"
+
+ - TESTS="smoke/test_diagnostics"


Please keep the name sorted, include your file under an existing test section/block, don't add a new job/runner.

Included test to the section with routers and ssvm tests.

No, you should keep it sorted by name as the file suggests.

- TESTS="smoke/test_accounts smoke/test_affinity_groups smoke/test_affinity_groups_projects smoke/test_deploy_vgpu_enabled_vm smoke/test_deploy_vm_iso smoke/test_deploy_vm_root_resize smoke/test_deploy_vm_with_userdata smoke/test_deploy_vms_with_varied_deploymentplanners **smoke/test_diagnostics** smoke/test_disk_offerings smoke/test_dynamicroles smoke/test_global_settings smoke/test_guest_vlan_range"

yadvr · 2018-07-10T10:49:10Z

 } else {
- throw new CloudRuntimeException("Command execution failed: " + details);
+ executionDetailsMap.put(ApiConstants.STDOUT, "");
+ executionDetailsMap.put(ApiConstants.STDERR, details );


Remove space after details.

…whitespace in Answer class

dhlaluku · 2018-07-10T16:55:10Z

@blueorangutan package

blueorangutan · 2018-07-10T16:55:48Z

@dhlaluku a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2018-07-10T17:26:20Z

Packaging result: ✔centos6 ✖centos7 ✔debian. JID-2168

yadvr · 2018-07-11T09:28:45Z

@blueorangutan package

blueorangutan · 2018-07-11T09:29:48Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2018-07-11T10:00:37Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2170

dhlaluku · 2018-07-11T10:01:09Z

@blueorangutan test

blueorangutan · 2018-07-11T10:01:48Z

@dhlaluku a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2018-07-11T20:16:58Z

Trillian test result (tid-2842)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35370 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2721-t2842-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_certauthority_root.py
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_public_ip_range.py
Intermitten failure detected: /marvin/tests/smoke/test_templates.py
Intermitten failure detected: /marvin/tests/smoke/test_usage.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_volumes.py
Intermitten failure detected: /marvin/tests/smoke/test_host_maintenance.py
Smoke tests completed. 60 look OK, 8 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_provision_certificate	`Error`	9.21	test_certauthority_root.py
ContextSuite context=TestDeployVirtioSCSIVM>:setup	`Error`	0.00	test_deploy_virtio_scsi_vm.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	1226.87	test_privategw_acl.py
test_04_extract_template	`Failure`	128.25	test_templates.py
ContextSuite context=TestISOUsage>:setup	`Error`	0.00	test_usage.py
test_01_secure_vm_migration	`Error`	5.14	test_vm_life_cycle.py
test_02_unsecure_vm_migration	`Error`	4.12	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	1.09	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	0.07	test_vm_life_cycle.py
test_06_download_detached_volume	`Failure`	137.56	test_volumes.py
test_11_migrate_volume_and_change_offering	`Error`	128.44	test_volumes.py
test_02_cancel_host_maintenace_with_migration_jobs	`Error`	2.24	test_host_maintenance.py

yadvr

LGTM

yadvr · 2018-07-13T11:26:32Z

Merging this based on code reviews and testing. The failures are not related to this PR.

This is a new feature for CS that allows Admin users improved troubleshooting of network issues in CloudStack hosted networks. Description: For troubleshooting purposes, CloudStack administrators may wish to execute network utility commands remotely on system VMs, or request system VMs to ping/traceroute/arping to specific addresses over specific interfaces. An API command to provide such functionalities is being developed without altering any existing APIs. The targeted system VMs for this feature are the Virtual Router (VR), Secondary Storage VM (SSVM) and the Console Proxy VM (CPVM). FS: https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+Remote+Diagnostics+API ML discussion: https://markmail.org/message/xt7owmb2c6iw7tva

Alteração da quantidade máxima de projetos através da UI Closesapache#2721 See merge request scclouds/scclouds!1193

dhlaluku changed the title ~~Introducing a new diagnostics API command for system VMs for CloudStack admins~~api:Introducing a new diagnostics API command for system VMs for CloudStack adminsJun 25, 2018

dhlaluku changed the title ~~api:Introducing a new diagnostics API command for system VMs for CloudStack admins~~api: Introducing a new diagnostics API command for system VMs for CloudStack adminsJun 25, 2018

borisstoyanov approved these changes Jun 26, 2018
View reviewed changes

yadvr requested review from DaanHoogland, nvazquez, rafaelweingartner and yadvr June 27, 2018 09:25

yadvr added the status:waiting-for-reviewer label Jun 27, 2018

yadvr added this to the 4.12.0.0 milestone Jun 27, 2018

DaanHoogland approved these changes Jul 3, 2018
View reviewed changes

yadvr reviewed Jul 4, 2018
View reviewed changes

yadvr approved these changes Jul 4, 2018
View reviewed changes

Dingane Hlaluku added 2 commits July 6, 2018 12:15

This is a new feature for CS that allows Admin users improved
102dce2
troubleshooting of network issues in CloudStack hosted networks

Add whitespace between operators
b9222e9

yadvr added the status:waiting-for-author label Jul 6, 2018

yadvr removed the status:waiting-for-reviewer label Jul 6, 2018

Dingane Hlaluku added 2 commits July 10, 2018 12:19

Add support for Simulator hypervisor
40ddc85

Format file against PEP-8
d2d0698

dhlaluku force-pushed the remote-diagnostics-api branch from bbe835d to d2d0698Compare July 10, 2018 10:43

yadvr reviewed Jul 10, 2018
View reviewed changes

Dingane Hlaluku added 2 commits July 10, 2018 13:08

Moving test file to already existing section, and removing redundant …
388dc36
…whitespace in Answer class

Moving test to the correct section following name grouping
a84bf42

yadvr removed the status:waiting-for-author label Jul 13, 2018

yadvr approved these changes Jul 13, 2018
View reviewed changes

yadvr merged commit 40af32b into apache:masterJul 13, 2018

dhlaluku deleted the remote-diagnostics-api branch July 17, 2018 12:00

dhlaluku mentioned this pull request Aug 31, 2018
Run diagnostics API UI integration for system VMs and VR #2833
Merged
12 tasks

bernardodemarco pushed a commit to scclouds/cloudstack that referenced this pull request Jul 16, 2025
Merge branch 'change-max-proj-ui' into '4.20.0.0-scclouds'
95225b5
Alteração da quantidade máxima de projetos através da UI Closesapache#2721 See merge request scclouds/scclouds!1193

		from nose.plugins.attrib import attr


		class TestRemoteDiagnostics(cloudstackTestCase):

Uh oh!

Conversation

dhlaluku commented Jun 25, 2018

Description

Types of changes

GitHub Issue/PRs

Screenshots (if appropriate):

How Has This Been Tested?

Checklist:

Uh oh!

borisstoyanov commented Jun 26, 2018

Uh oh!

blueorangutan commented Jun 26, 2018

Uh oh!

borisstoyanov left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Jun 26, 2018

Uh oh!

borisstoyanov commented Jun 26, 2018

Uh oh!

blueorangutan commented Jun 26, 2018

Uh oh!

blueorangutan commented Jun 26, 2018

Uh oh!

borisstoyanov commented Jun 27, 2018

Uh oh!

blueorangutan commented Jun 27, 2018

Uh oh!

borisstoyanov commented Jun 28, 2018

Uh oh!

blueorangutan commented Jun 28, 2018

Uh oh!

blueorangutan commented Jun 28, 2018

Uh oh!

borisstoyanov commented Jul 2, 2018

Uh oh!

blueorangutan commented Jul 2, 2018

Uh oh!

blueorangutan commented Jul 2, 2018

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yadvr left a comment

Choose a reason for hiding this comment

Uh oh!

yadvr commented Jul 6, 2018

Uh oh!

yadvr commented Jul 10, 2018

Uh oh!

dhlaluku commented Jul 10, 2018

Uh oh!

yadvr commented Jul 10, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dhlaluku commented Jul 10, 2018

Uh oh!

blueorangutan commented Jul 10, 2018

Uh oh!

blueorangutan commented Jul 10, 2018

Uh oh!

yadvr commented Jul 11, 2018

Uh oh!

borisstoyanov left a comment •
edited
Loading