Stop writing to '/var/log/app_engine/' and write logs to Stackdriver logging API by liyanhui1228 · Pull Request #3410 · googleapis/google-cloud-python

liyanhui1228 · 2017-05-12T18:46:38Z

For #2997 . Modified the AppEngineHandler to extend the CloudLoggingHandler and directly write logs to Stackdriver Logging API.

Screenshot of the gae_app log with AppEngineHandler:

Sign in to view

+ transport=BackgroundThreadTransport,
+ resource=_GLOBAL_RESOURCE):
+ super(AppEngineHandler, self).__init__(client, name, transport)
+ self.resource=resource


Sign in to view

 self.stopped = True

- def enqueue(self, record, message):
+ def enqueue(self, record, message, resource=_GLOBAL_RESOURCE):


Sign in to view


-class AppEngineHandler(logging.handlers.RotatingFileHandler):
- """A handler that writes to the App Engine fluentd Stackdriver log file.
+_GLOBAL_RESOURCE = Resource(type='global', labels={})


Sign in to view

-_LOG_FILE_COUNT = 3
+DEFAULT_LOGGER_NAME = 'python'

+EXCLUDED_LOGGER_DEFAULTS = ('google.cloud', 'oauth2client')


Sign in to view

-_LOG_PATH_TEMPLATE = '/var/log/app_engine/app.{pid}.json'
-_MAX_LOG_BYTES = 128 * 1024 * 1024
-_LOG_FILE_COUNT = 3
+DEFAULT_LOGGER_NAME = 'python'


Sign in to view


-class AppEngineHandler(logging.handlers.RotatingFileHandler):
- """A handler that writes to the App Engine fluentd Stackdriver log file.
+_GLOBAL_RESOURCE = Resource(type='global', labels={})


Sign in to view

-_LOG_FILE_COUNT = 3
+DEFAULT_LOGGER_NAME = 'python'

+EXCLUDED_LOGGER_DEFAULTS = ('google.cloud', 'oauth2client')


Sign in to view

+ transport=BackgroundThreadTransport,
+ resource=_GLOBAL_RESOURCE):
+ super(AppEngineHandler, self).__init__(client, name, transport)
+ self.resource=resource


Sign in to view

- message = super(AppEngineHandler, self).format(record)
-return format_stackdriver_json(record, message)
+ message = super(CloudLoggingHandler, self).format(record)
+self.transport.send(record, message, self.resource)


Sign in to view

 self.stopped = True

- def enqueue(self, record, message):
+ def enqueue(self, record, message, resource=_GLOBAL_RESOURCE):


liyanhui1228 · 2017-05-13T00:17:20Z

Ready for another review, but still wait for Bill's comment about the default logName.

waprin · 2017-05-13T01:22:12Z

You are right that shutdown part of logName is not quite right, I copied and pasted it from somewhere else. I am not sure exactly what the logName should be to be honest, I think it doesn't matter that much because the resource and labels are what puts it in the right place. I would guess its "projects/coinflow-demo/logs/appengine.googleapis.com%2Fapp" ? Might be worth pinging AppEngine or Logging people who can answer more confidently since I'm just guessing based on what I've seen a few other places.

theacodes · 2017-05-13T01:26:30Z

I wonder what happens if you just name it "app".

…

On Fri, May 12, 2017, 6:22 PM Bill Prin ***@***.***> wrote: You are right that shutdown part of logName is not quite right, I copied and pasted it from somewhere else. I am not sure exactly what the logName should be to be honest, I think it doesn't matter that much because the resource and labels are what puts it in the right place. I would guess its "projects/coinflow-demo/logs/appengine.googleapis.com%2Fapp" ? Might be worth pinging AppEngine or Logging people who can answer more confidently since I'm just guessing based on what I've seen a few other places. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3410 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUcx213jZC7dp-NC5vYNeKOtQqcPXQks5r5QXFgaJpZM4NZpk_> .

tmatsuo · 2017-05-13T06:48:00Z

FWIW, I'd recommend that you use app for the default logName. The logs with the correct monitored resource and if the logName is "projects/PROJECT_ID/logs/app", then it's available as "app" in the Logging UI drop down menu. PHP logging library is using app as the default logName.

Sign in to view

-_LOG_FILE_COUNT = 3
+DEFAULT_LOGGER_NAME = 'python'

+EXCLUDED_LOGGER_DEFAULTS = ('google.cloud', 'oauth2client')


Sign in to view

+GAE_RESOURCE = Resource(
+ type='gae_app',
+ labels={
+ 'project_id': os.getenv('GCLOUD_PROJECT'),


Sign in to view

+ def __init__(self, client,
+ name=DEFAULT_LOGGER_NAME,
+ transport=BackgroundThreadTransport,
+ resource=GAE_RESOURCE):


Sign in to view

-_LOG_PATH_TEMPLATE = '/var/log/app_engine/app.{pid}.json'
-_MAX_LOG_BYTES = 128 * 1024 * 1024
-_LOG_FILE_COUNT = 3
+DEFAULT_LOGGER_NAME = 'python'


Sign in to view

 self.logger = client.logger(name)

- def send(self, record, message):
+ def send(self, record, message, resource=_GLOBAL_RESOURCE):


Sign in to view

+ def test_ctor(self):
+ client = _Client(self.PROJECT)
+ handler = self._make_one(client, transport=_Transport)
+ self.assertEqual(handler.client, client)


Sign in to view

 self.assertEqual(logger.name, name)

 def test_send(self):
+ RESOURCE = Resource(


Sign in to view

 self.assertEqual(transport.logger.name, 'python_logger')

 def test_send(self):
+ RESOURCE = Resource(


Sign in to view


 import logging
 import unittest
+from google.cloud.logging.resource import Resource


…logging API

Sign in to view


-Logs to the well-known file that the fluentd sidecar container on App Engine
-Flexible is configured to read from and send to Stackdriver Logging.
+Send logs to Stackdriver Logging API.


Sign in to view

-# This file is largely copied from:
-# https://github.com/GoogleCloudPlatform/python-compat-runtime/blob/master
-# /appengine-vmruntime/vmruntime/cloud_logging.py
+GAE_PROJECT_ENV = 'GCLOUD_PROJECT'


Sign in to view

- message = super(AppEngineHandler, self).format(record)
- return format_stackdriver_json(record, message)
+ def __init__(self, client,
+ name=EXCLUDED_LOGGER_DEFAULTS,


Sign in to view

 self.name = name
 self.client = client
 self.transport = transport(client, name)
+ self.resource=resource


Sign in to view

 """
 message = super(CloudLoggingHandler, self).format(record)
- self.transport.send(record, message)
+ self.transport.send(record, message, self.resource)


Sign in to view

+ to the global resource type.
 """
- self.worker.enqueue(record, message)
+ self.worker.enqueue(record, message, resource)


Sign in to view

+ labels={
+ 'project_id': os.environ.get(GAE_PROJECT_ENV),
+ 'module_id': os.environ.get('GAE_SERVICE'),
+ 'version_id': os.environ.get('GAE_VERSION'),


Sign in to view

- PROJECT = 'PROJECT'
+ from google.cloud.logging.resource import Resource
+
+ GAE_PROJECT_ENV = 'PROJECT'


Sign in to view

+ client = _Client(self.GAE_PROJECT_ENV)
+ handler = self._make_one(client, transport=_Transport)
+ self.assertEqual(handler.client, client)
+ self.assertEqual(handler.resource, self.GAE_RESOURCE)


tmatsuo · 2017-05-16T06:33:29Z

Re: default logName

I recommended that we use app, but actually it can be anything. The request correlation feature just works with any logName as long as they have the same monitoredResource and the traceId. Both app and python are natural choice (shutdown is not though).

theacodes

Looks almost good to me, @dhermes can you take a look?

Sign in to view

-class AppEngineHandler(logging.handlers.RotatingFileHandler):
- """A handler that writes to the App Engine fluentd Stackdriver log file.
+class AppEngineHandler(CloudLoggingHandler):
+ """A handler that directly makes Stackdriver logging API calls.


Sign in to view

-
- def __init__(self):
- """Construct the handler
+ This handler can be used to route Python standard logging messages directly


Sign in to view

- super(AppEngineHandler, self).__init__(self.filename,
- maxBytes=_MAX_LOG_BYTES,
- backupCount=_LOG_FILE_COUNT)
+ This handler supports both an asynchronous and synchronous transport.


Sign in to view

+ :param transport: Class for creating new transport objects. It should
+ extend from the base :class:`.Transport` type and
+ implement :meth`.Transport.send`. Defaults to
+ :class:`.BackgroundThreadTransport`. The other


Sign in to view

- """
- message = super(AppEngineHandler, self).format(record)
- return format_stackdriver_json(record, message)
+ DEFAULT_LOGGER_NAME = 'app'


Sign in to view

+
+ def __init__(self, client,
+ transport=BackgroundThreadTransport):
+ super(AppEngineHandler, self).__init__(client, name=self.DEFAULT_LOGGER_NAME,


Sign in to view

+ transport=transport, resource=self.gae_resource)
+
+ @property
+ def gae_resource(self):


googlebot · 2017-05-16T23:46:54Z

CLAs look good, thanks!

liyanhui1228 · 2017-05-17T18:33:24Z

@dhermes @jonparrott Ready for another review or merge.

dhermes · 2017-05-17T18:35:17Z

I just re-built CircleCI, let's get it green (I realize the failure was due to error_reporting)

liyanhui1228 · 2017-05-17T18:37:54Z

OK!

Sign in to view

 """
- message = super(AppEngineHandler, self).format(record)
- return format_stackdriver_json(record, message)
+


Sign in to view

 option is :class:`.SyncTransport`.

+ :type resource: :class:`~google.cloud.logging.resource.Resource`
+ :param resource: (Optional) Monitored resource of the entry


Sign in to view

- from google.cloud._testing import _Monkey
- from google.cloud.logging.handlers import app_engine as _MUT
+ def test_constructor(self):
+ import mock


Sign in to view

+class _Transport(object):
+
+ def __init__(self, client, name):
+ pass


Sign in to view



 class _Batch(object):
+ from google.cloud.logging.logger import _GLOBAL_RESOURCE


Sign in to view

- def log_struct(self, info, severity=logging.INFO):
- self.log_struct_called_with = (info, severity)
+ def log_struct(self, info, severity=logging.INFO, resource=_GLOBAL_RESOURCE):
+ self.log_struct_called_with = (info, severity, resource)


dhermes · 2017-05-17T19:07:03Z

@liyanhui1228 Looks like you --force pushed over some changes I made yesterday? Pre #3422, the build will fail, so I rebased your branch. There was also a tiny docs error that I made a commit to resolve (error doesn't seem to be present any longer, just ran docs locally).

liyanhui1228 · 2017-05-17T20:22:00Z

Yeah did you rebase my branch just now? I didn't find it in git log.

dhermes · 2017-05-17T20:42:10Z

@liyanhui1228 No I didn't do it just now, but a rebase won't show up in your logs (other than the fact that all the commit hashes will be different and the first commit will have a new parent)

dhermes · 2017-05-17T20:55:57Z

This LGTM. We should wait until we can get a green build? WDYT @jonparrott?

theacodes · 2017-05-17T20:57:39Z

We should wait until we can get a green build?

Yes. I'll give it another review while we wait for CI.

theacodes · 2017-05-17T21:30:43Z

@dhermes bigquery system tests are failing?

theacodes · 2017-05-17T22:15:13Z

Confirmed with @dhermes over chat that bigquery tests are borked - merging based on local verification of tests.

dhermes · 2017-05-17T22:15:40Z

They are green sometimes but I've seen them fail 4+ times on this PR.

tseaver · 2017-05-19T20:24:10Z

@jonparrott It looks like you merged this PR with failing coverage in the logging package:

$ cd logging
$ nox
...
Name Stmts Miss Branch BrPart Cover Missing
-----------------------------------------------------------------------------------------------------------
google/cloud/logging/__init__.py 7 0 0 0 100%
google/cloud/logging/_gax.py 190 0 28 0 100%
google/cloud/logging/_helpers.py 12 0 6 0 100%
google/cloud/logging/_http.py 98 0 16 0 100%
google/cloud/logging/client.py 79 0 20 0 100%
google/cloud/logging/entries.py 57 0 10 0 100%
google/cloud/logging/handlers/__init__.py 6 0 0 0 100%
google/cloud/logging/handlers/_helpers.py 7 0 0 0 100%
google/cloud/logging/handlers/app_engine.py 15 0 0 0 100%
google/cloud/logging/handlers/container_engine.py 7 0 0 0 100%
google/cloud/logging/handlers/handlers.py 26 0 2 0 100%
google/cloud/logging/handlers/transports/__init__.py 5 0 0 0 100%
google/cloud/logging/handlers/transports/background_thread.py 98 0 24 0 100%
google/cloud/logging/handlers/transports/base.py 5 0 0 0 100%
google/cloud/logging/handlers/transports/sync.py 8 0 0 0 100%
google/cloud/logging/logger.py 116 0 50 0 100%
google/cloud/logging/metric.py 46 0 2 0 100%
google/cloud/logging/resource.py 8 0 0 0 100%
google/cloud/logging/sink.py 46 0 2 0 100%
tests/unit/__init__.py 0 0 0 0 100%
tests/unit/handlers/__init__.py 0 0 0 0 100%
tests/unit/handlers/test_app_engine.py 40 0 0 0 100%
tests/unit/handlers/test_container_engine.py 19 0 0 0 100%
tests/unit/handlers/test_handlers.py 60 0 2 0 100%
tests/unit/handlers/transports/__init__.py 0 0 0 0 100%
tests/unit/handlers/transports/test_background_thread.py 200 0 4 1 99% 293->296
tests/unit/handlers/transports/test_base.py 15 0 0 0 100%
tests/unit/handlers/transports/test_sync.py 42 0 0 0 100%
tests/unit/test__gax.py 989 0 48 0 100%
tests/unit/test__helpers.py 32 0 0 0 100%
tests/unit/test__http.py 481 0 2 0 100%
tests/unit/test_client.py 375 0 0 0 100%
tests/unit/test_entries.py 190 0 0 0 100%
tests/unit/test_logger.py 506 0 0 0 100%
tests/unit/test_metric.py 152 0 0 0 100%
tests/unit/test_sink.py 153 0 0 0 100%
-----------------------------------------------------------------------------------------------------------
TOTAL 4090 0 216 1 99%
nox > Command coverage report --show-missing --fail-under=100 failed with exit code 2
nox > Session cover failed. :(
nox > Ran multiple sessions:
nox >* unit_tests(python_version='2.7'): passed
nox >* unit_tests(python_version='3.4'): passed
nox >* unit_tests(python_version='3.5'): passed
nox >* unit_tests(python_version='3.6'): passed
nox >* system_tests(python_version='2.7'): passed
nox >* system_tests(python_version='3.6'): passed
nox >* lint: passed
nox >* lint_setup_py: passed
nox >* cover: failed

We need to quit running tests for the unaffected packages ASAP. Having BQ back-end slow down shouldn't break our logging CI.

theacodes · 2017-05-19T20:38:17Z

Agreed. I'll send a PR to address coverage when I return from pycon next week.

…

On Fri, May 19, 2017, 1:24 PM Tres Seaver ***@***.***> wrote: @jonparrott <https://github.com/jonparrott> It looks like you merged this PR with failing coverage *in the logging package*: $ cd logging $ nox ... Name Stmts Miss Branch BrPart Cover Missing ----------------------------------------------------------------------------------------------------------- google/cloud/logging/__init__.py 7 0 0 0 100% google/cloud/logging/_gax.py 190 0 28 0 100% google/cloud/logging/_helpers.py 12 0 6 0 100% google/cloud/logging/_http.py 98 0 16 0 100% google/cloud/logging/client.py 79 0 20 0 100% google/cloud/logging/entries.py 57 0 10 0 100% google/cloud/logging/handlers/__init__.py 6 0 0 0 100% google/cloud/logging/handlers/_helpers.py 7 0 0 0 100% google/cloud/logging/handlers/app_engine.py 15 0 0 0 100% google/cloud/logging/handlers/container_engine.py 7 0 0 0 100% google/cloud/logging/handlers/handlers.py 26 0 2 0 100% google/cloud/logging/handlers/transports/__init__.py 5 0 0 0 100% google/cloud/logging/handlers/transports/background_thread.py 98 0 24 0 100% google/cloud/logging/handlers/transports/base.py 5 0 0 0 100% google/cloud/logging/handlers/transports/sync.py 8 0 0 0 100% google/cloud/logging/logger.py 116 0 50 0 100% google/cloud/logging/metric.py 46 0 2 0 100% google/cloud/logging/resource.py 8 0 0 0 100% google/cloud/logging/sink.py 46 0 2 0 100% tests/unit/__init__.py 0 0 0 0 100% tests/unit/handlers/__init__.py 0 0 0 0 100% tests/unit/handlers/test_app_engine.py 40 0 0 0 100% tests/unit/handlers/test_container_engine.py 19 0 0 0 100% tests/unit/handlers/test_handlers.py 60 0 2 0 100% tests/unit/handlers/transports/__init__.py 0 0 0 0 100% tests/unit/handlers/transports/test_background_thread.py 200 0 4 1 99% 293->296 tests/unit/handlers/transports/test_base.py 15 0 0 0 100% tests/unit/handlers/transports/test_sync.py 42 0 0 0 100% tests/unit/test__gax.py 989 0 48 0 100% tests/unit/test__helpers.py 32 0 0 0 100% tests/unit/test__http.py 481 0 2 0 100% tests/unit/test_client.py 375 0 0 0 100% tests/unit/test_entries.py 190 0 0 0 100% tests/unit/test_logger.py 506 0 0 0 100% tests/unit/test_metric.py 152 0 0 0 100% tests/unit/test_sink.py 153 0 0 0 100% ----------------------------------------------------------------------------------------------------------- TOTAL 4090 0 216 1 99% nox > Command coverage report --show-missing --fail-under=100 failed with exit code 2 nox > Session cover failed. :( nox > Ran multiple sessions: nox > * unit_tests(python_version='2.7'): passed nox > * unit_tests(python_version='3.4'): passed nox > * unit_tests(python_version='3.5'): passed nox > * unit_tests(python_version='3.6'): passed nox > * system_tests(python_version='2.7'): passed nox > * system_tests(python_version='3.6'): passed nox > * lint: passed nox > * lint_setup_py: passed nox > * cover: failed We need to quit running tests for the unaffected packages ASAP. Having BQ back-end slow down shouldn't break our logging CI. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3410 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUc6L81Rgawf1tF7Fqka6T1kPzbJgcks5r7fprgaJpZM4NZpk_> .

tseaver · 2017-05-19T20:40:38Z

I just merged #3445, which fixes the immediate issue (by suppressing the uncovered branch nested down in the tests).

theacodes · 2017-05-19T20:43:01Z

Awesome, thank you.

…

On Fri, May 19, 2017, 1:40 PM Tres Seaver ***@***.***> wrote: I just merged #3445 <#3445>, which fixes the immediate issue (by suppressing the uncovered branch nested down in the tests). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3410 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAPUcwyvcil1FXi9vUAPdiJ1vjpaGRpKks5r7f5HgaJpZM4NZpk_> .

…tform/python-docs-samples#3410) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-cloud-kms](https://togithub.com/googleapis/python-kms) | minor | `==1.3.0` -> `==1.4.0` | --- ### Release Notes <details> <summary>googleapis/python-kms</summary> ### [`v1.4.0`](https://togithub.com/googleapis/python-kms/blob/master/CHANGELOG.md#140-httpswwwgithubcomgoogleapispython-kmscomparev130v140-2020-04-14) [Compare Source](https://togithub.com/googleapis/python-kms/compare/v1.3.0...v1.4.0) ##### Features - add support for external key manager (via synth) ([#8](https://www.github.com/googleapis/python-kms/issues/8)) ([4077fc8](https://www.github.com/googleapis/python-kms/commit/4077fc89943cc09d489d44c05efcf9cab61cdbaf)) </details> --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples).

…logging API (#3410)

liyanhui1228 requested review from dhermes, duggelz, theacodes and waprin May 12, 2017 18:46

googlebot added the cla: yes This human has signed the Contributor License Agreement. label May 12, 2017

waprin reviewed May 12, 2017
View reviewed changes

theacodes suggested changes May 12, 2017
View reviewed changes

liyanhui1228 force-pushed the yanhuil/2997 branch 2 times, most recently from f990734 to 5b79d48Compare May 13, 2017 00:15

theacodes suggested changes May 15, 2017
View reviewed changes

tseaver added the api: logging Issues related to the Cloud Logging API. label May 15, 2017

liyanhui1228 added 2 commits May 15, 2017 12:45

Stop writing to '/var/log/app_engine/' and write logs to Stackdriver …
50bf434
…logging API

Address jon and bill's comments
4d1924f

liyanhui1228 force-pushed the yanhuil/2997 branch from 5b79d48 to c4704d6Compare May 15, 2017 21:50

theacodes suggested changes May 15, 2017
View reviewed changes

liyanhui1228 force-pushed the yanhuil/2997 branch from c4704d6 to 4172e6dCompare May 16, 2017 00:15

fix stuff
2591ef1

liyanhui1228 force-pushed the yanhuil/2997 branch from 4172e6d to 2591ef1Compare May 16, 2017 00:28

Change default logName
38f62d9

theacodes suggested changes May 16, 2017
View reviewed changes

liyanhui1228 force-pushed the yanhuil/2997 branch from d43ae32 to d2d9628Compare May 16, 2017 20:45

googlebot added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels May 16, 2017

duggelz removed their request for review May 17, 2017 18:40

dhermes reviewed May 17, 2017
View reviewed changes

fix style
1b3d91e

theacodes approved these changes May 17, 2017
View reviewed changes

theacodes merged commit cce9c99 into masterMay 17, 2017

theacodes deleted the yanhuil/2997 branch May 17, 2017 22:15

This was referenced May 17, 2017
Send trace context with logs from web applications #3359
Closed
Stop writing to /var/log/app_engine/#2997
Closed

tseaver mentioned this pull request Jun 26, 2017
Prep logging-1.1.0 release. #3528
Merged

bonfante mentioned this pull request Jul 10, 2018
There is no existing directory at "/var/log/app_engine" and its not buildab GoogleCloudPlatform/php-docker#452
Open

parthea pushed a commit that referenced this pull request Nov 24, 2025
Stop writing to '/var/log/app_engine/' and write logs to Stackdriver …
54395b9
…logging API (#3410)



		class _Batch(object):
		from google.cloud.logging.logger import _GLOBAL_RESOURCE

Uh oh!

Conversation

liyanhui1228 commented May 12, 2017• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

liyanhui1228 commented May 13, 2017

Uh oh!

waprin commented May 13, 2017

Uh oh!

theacodes commented May 13, 2017 via email

Uh oh!

tmatsuo commented May 13, 2017

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

liyanhui1228 commented May 12, 2017•
edited
Loading

tmatsuo commented May 16, 2017•
edited
Loading