Add GAE and GKE fluentd Handlers by waprin · Pull Request #2668 · googleapis/google-cloud-python

waprin · 2016-11-03T00:22:27Z

On GAE and GKE with the plugin installed, there is a fluentd plugin that collects logs from files. However without the right formatting, metadata like log_level is lost. Furthermore, the fluentd agents are configured to set the correct resources types, which could be done in the main handler as well, but it’s easier to rely on the fluentd configurations.

This adds two new handlers and some helper functions to detect when they should be used.

waprin · 2016-11-03T00:23:30Z

@dhermes /cc @jonparrott

daspecster

Pretty awesome stuff!

Sign in to view

@@ -0,0 +1,6 @@
+Google App Engine Flexible Log Handler
+=======================================


Sign in to view

@@ -0,0 +1,6 @@
+Google Container Engine Log Handler
+=======================================


Sign in to view

-Logging client.
+It's possible to tie the Python :mod:`logging` module directly into Google Stackdriver Logging.
+There are different handler options to accomplish this. To automatically pick the default,
+use :meth:``google.cloud.logging.client.get_default_handler`.


Sign in to view

 >>> cloud_logger.error('bad news')

+It is also possible to attach the handler to the root Python logger, so that for example a plain
+`logging.warn` call would be sent to Cloud Logging, as well as any other loggers created. However,


Sign in to view

-method :meth:`setup_logging <google.cloud.logging.handlers.setup_logging>` is provided to configure
-this automatically:
+CloudLoggingHandler
+=======================


Sign in to view

 """Python :mod:`logging` handlers for Google Cloud Logging."""

 from google.cloud.logging.handlers.handlers import CloudLoggingHandler
+from google.cloud.logging.handlers.GAEHandler import GAEHandler


Sign in to view

- :param excluded_loggers: The loggers to not attach the handler to. This
- will always include the loggers in the path of
- the logging client itself.
+ :param excluded_loggers: (Optional)The loggers to not attach the handler


Sign in to view

+ to. This will always include the loggers in the
+ path of the logging client itself.
+
+ :type log_level: :mod:`logging` log level


Sign in to view

 class _Client(object):

- def __init__(self, project):
+ def __init__(self, project, _=None, __=None):


Sign in to view

 return self

+ def authorize(self, http):
+ pass


Sign in to view

+Logs to the well-known file that the fluentd sidecar container on App Engine
+Flexible is configured to read from and send to Stackdriver Logging.
+
+This file is largely copied from:


Sign in to view

+import math
+import os
+
+LOG_PATH_TEMPLATE = '/var/log/app_engine/app.{pid}.json'


Sign in to view

+LOG_FILE_COUNT = 3
+
+
+class GAEHandler(logging.handlers.RotatingFileHandler):


Sign in to view

+
+
+class GAEHandler(logging.handlers.RotatingFileHandler):
+ """A handler that writes to the GAE fluentd Stackdriver logging file.


Sign in to view

+class GAEHandler(logging.handlers.RotatingFileHandler):
+ """A handler that writes to the GAE fluentd Stackdriver logging file.
+
+ Writes to the file that the fluentd agent on App Engine Flexbile is


Sign in to view

-target=self._run,
- name='google.cloud.logging.handlers.transport.Worker')
+thread_name = 'google.cloud.logging.handlers.transport.Worker'
+self._thread = threading.Thread(target=self._run,


Sign in to view

+ LOGNAME = 'loggername'
+ MESSAGE = 'hello world'
+ record = _Record(LOGNAME, logging.INFO, MESSAGE)
+ expected_payload = {'message': MESSAGE,


Sign in to view

+ self.assertEqual(payload, json.dumps(expected_payload))
+
+
+class _Record(object):


Sign in to view

+ import logging
+ import json
+ handler = self._makeOne()
+ LOGNAME = 'loggername'


Sign in to view

 class _Client(object):

- def __init__(self, project):
+ def __init__(self, project, _=None, __=None):


Sign in to view

 logging-sink
 logging-handlers
+ logging-handlers-GAEHandler
+ logging-handlers-GKEHandler


Sign in to view

+Google App Engine Flexible Log Handler
+=======================================
+
+.. automodule:: google.cloud.logging.handlers.GAEHandler


Sign in to view

+Google Container Engine Log Handler
+=======================================
+
+.. automodule:: google.cloud.logging.handlers.GKEHandler


Sign in to view

-create a :class:`CloudLoggingHandler <google.cloud.logging.CloudLoggingHandler>` instance from your
-Logging client.
+It's possible to tie the Python :mod:`logging` module directly into Google Stackdriver Logging.
+There are different handler options to accomplish this. To automatically pick the default,


Sign in to view

-Logging client.
+It's possible to tie the Python :mod:`logging` module directly into Google Stackdriver Logging.
+There are different handler options to accomplish this. To automatically pick the default,
+use :meth:``google.cloud.logging.client.get_default_handler`.


Sign in to view

+
+ client = self._makeOne(self.PROJECT, credentials=_Credentials())
+
+ with _Monkey(os, getenv=lambda var: var == 'GAE_APPENGINE_HOSTNAME'):


Sign in to view

+
+ self.assertIsInstance(handler, GKEHandler)
+
+ with _Monkey(os, getenv=lambda _: None):


Sign in to view

+
+ self.assertIsInstance(handler, GAEHandler)
+
+ with _Monkey(os, getenv=lambda var: var == 'KUBERNETES_SERVICE'):


Sign in to view

+ def test_setup_logging(self):
+ from google.cloud._testing import _Monkey
+ import google.cloud.logging.client as MUT
+ client = self._makeOne(


Sign in to view

+ self.called = False
+
+ def __call__(self, _, log_level,
+ excluded_loggers=None): # pylintdisable=unused-variable


waprin · 2016-11-04T00:09:33Z

Think I addressed all comments, let me make sure Travis is passing and then I'll do another once-over before I request another review.

Sign in to view


 Integration with Python logging module
---------------------------------------------
+---------------------------------------


Sign in to view

 credentials=_Credentials(),
 use_gax=False)
- os.environ[_CONTAINER_ENGINE_ENV] = True
+ os.environ[_CONTAINER_ENGINE_ENV] = "True"


Sign in to view

+(i.e. your app.yaml contains ``runtime: python``), and
+:class:`~google.cloud.handlers.container_engine.ContainerEngineHandler`,
+which is
+recommended when running on `Google Container Engine` with the Stackdriver


Sign in to view

+from google.cloud.logging.handlers import AppEngineHandler
+from google.cloud.logging.handlers import ContainerEngineHandler
+from google.cloud.logging.handlers import setup_logging
+from google.cloud.logging.handlers.handlers import EXCLUDED_LOGGER_DEFAULTS


Sign in to view

+"""
+
+# This file is largely copied from:
+# https://github.com/GoogleCloudPlatform/python-compat-runtime/blob/master/appengine-vmruntime/vmruntime/cloud_logging.py


Sign in to view

- >>> setup_logging(handler, excluded_loggers=('werkzeug',)))
+ >>> handler = CloudLoggingHandler(client, name="mycustomlog")
+
+fluentd logging handlers


Sign in to view

+the environment to automatically detect whether the code is running in
+these platforms and use the appropriate handler.
+
+In both cases, the fluentd agent is configured to automatically parse log files


Sign in to view

-method :meth:`setup_logging <google.cloud.logging.handlers.setup_logging>` is provided to configure
-this automatically:
+If you prefer not to use :meth:`google.cloud.logging.client
+.get_default_handler`, you can directly create a


Sign in to view

+ 'message': message,
+ 'timestamp': {
+ 'seconds': 5,
+ 'nanos': int(.03 * 1e9)


Sign in to view

+ 'timestamp': {'seconds': 5,
+ 'nanos': int(.03 * 1e9)},
+ 'thread': record.thread,
+ 'severity': record.levelname}


Sign in to view

+ 'nanos': int(subsecond * 1e9)
+ },
+ 'thread': record.thread,
+ 'severity': record.levelname


Sign in to view

+ 'message': message,
+ 'timestamp': {
+ 'seconds': int(second),
+ 'nanos': int(subsecond * 1e9)


Sign in to view

+ payload = {
+ 'message': message,
+ 'timestamp': {'seconds': int(second),
+ 'nanos': int(subsecond * 1e9)},


Sign in to view

+class AppEngineHandler(logging.handlers.RotatingFileHandler):
+ """A handler that writes to the App Engine fluentd Stackdriver log file.
+
+ Writes to the file that the fluentd agent on App Engine Flexbile is


Sign in to view

+ itself, and not as a formatting step, so as not to interfere with
+ user-defined logging formats.
+
+ Logging calls can also alternatively 'trace_id' in as a field in the


Sign in to view

+"""Logging handler for Google Container Engine (GKE).
+
+Formats log messages in a JSON format, so that Kubernetes clusters with the
+fluentd google cloud plugin installed can format their log messages so that


Sign in to view

+
+
+class ContainerEngineHandler(logging.StreamHandler):
+ """Handler to formats log messages in good fluentd GKE format.


Sign in to view

 """Python :mod:`logging` handlers for Google Cloud Logging."""

 import logging
-


Sign in to view

+class TestContainerEngineHandler(unittest.TestCase):
+ PROJECT = 'PROJECT'
+
+ def _getTargetClass(self):


Sign in to view

+ 'message': message,
+ 'timestamp': {
+ 'seconds': 5,
+ 'nanos': int(.03 * 1e9)


Sign in to view

- 'message': MESSAGE,
- 'python_logger': PYTHON_LOGGER_NAME
+ 'message': message,
+ 'python_logger': python_logger_name


Sign in to view

- 'message': MESSAGE,
- 'python_logger': PYTHON_LOGGER_NAME
+ 'message': message,
+ 'python_logger': python_logger_name


Sign in to view

 return self

+ def authorize(self, http):
+ pass


Sign in to view

 self._scopes = scope
 return self

+ def authorize(self, _):


Sign in to view

 # See the License for the specific language governing permissions and
 # limitations under the License.

+import mock


Sign in to view

 use_gax=False)
 handler = client.get_default_handler()
 self.assertIsInstance(handler, CloudLoggingHandler)
+ self.assertTrue(credentials.authorized, http_mock)


Sign in to view

+ def test_get_default_handler_general(self, deepcopy):
 from google.cloud.logging.handlers import CloudLoggingHandler

+ http_mock = mock.Mock()


Sign in to view

 from google.cloud._testing import _Monkey
 import google.cloud.logging.client as MUT

+ http_mock = mock.Mock()


Sign in to view


- def test_setup_logging(self):
+ @mock.patch('copy.deepcopy')
+ def test_setup_logging(self, deepcopy):


dhermes · 2016-11-21T22:13:28Z

@waprin Pretty please squash (either in the GitHub UI or in your local copy).

@jonparrott Do you want to do a final pass?

On GAE and GKE with the plugin installed, there is a fluentd plugin that collects logs from files. However without the right formatting, metadata like log_level is lost. Furthermore, the fluentd agents are configured to set the correct resources types, which could be done in the main handler as well, but it’s easier to rely on the fluentd configurations. This adds two new handlers and some helper functions to detect when they should be used.

Add GAE and GKE fluentd Handlers

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Nov 3, 2016

daspecster reviewed Nov 3, 2016
View reviewed changes

theacodes suggested changes Nov 3, 2016
View reviewed changes

theacodes self-assigned this Nov 3, 2016

dhermes reviewed Nov 3, 2016
View reviewed changes

This was referenced Nov 3, 2016
Python Logging API always defaults to Custom logs #2673
Closed
Convert logging doctest examples to testable snippets. #2675
Merged

dhermes reviewed Nov 4, 2016
View reviewed changes

waprin force-pushed the newhandlers branch from a81e8dd to 31641bdCompare November 4, 2016 20:56

dhermes reviewed Nov 4, 2016
View reviewed changes

waprin force-pushed the newhandlers branch 4 times, most recently from c755a71 to c132584Compare November 16, 2016 21:50

dhermes reviewed Nov 18, 2016
View reviewed changes

Comment threadlogging/unit_tests/test_client.py Outdated
self._scopes = scope
return self

def authorize(self, _):
This comment was marked as spam.
Sign in to view

dhermes reviewed Nov 21, 2016
View reviewed changes

dhermes approved these changes Nov 21, 2016
View reviewed changes

waprin force-pushed the newhandlers branch from 4477d59 to a65ef9fCompare November 21, 2016 22:21

waprin force-pushed the newhandlers branch from a65ef9f to 02ae299Compare November 21, 2016 22:24

theacodes approved these changes Nov 21, 2016
View reviewed changes

dhermes merged commit 5d59291 into googleapis:masterNov 22, 2016

daspecster added the api: logging Issues related to the Cloud Logging API. label Nov 22, 2016

dhermes mentioned this pull request Dec 9, 2016
Update versions for mega-release. #2846
Merged

richkadel pushed a commit to richkadel/google-cloud-python that referenced this pull request May 6, 2017
Merge pull requestgoogleapis#2668from waprin/newhandlers
a787604
Add GAE and GKE fluentd Handlers

parthea pushed a commit that referenced this pull request Nov 24, 2025
Merge pull request#2668from waprin/newhandlers
392f14c
Add GAE and GKE fluentd Handlers

		@@ -0,0 +1,6 @@
		Google App Engine Flexible Log Handler
		=======================================

		@@ -0,0 +1,6 @@
		Google Container Engine Log Handler
		=======================================

		LOG_FILE_COUNT = 3


		class GAEHandler(logging.handlers.RotatingFileHandler):



		class GAEHandler(logging.handlers.RotatingFileHandler):
		"""A handler that writes to the GAE fluentd Stackdriver logging file.

		self.assertEqual(payload, json.dumps(expected_payload))


		class _Record(object):


		client = self._makeOne(self.PROJECT, credentials=_Credentials())

		with _Monkey(os, getenv=lambda var: var == 'GAE_APPENGINE_HOSTNAME'):


		self.assertIsInstance(handler, GKEHandler)

		with _Monkey(os, getenv=lambda _: None):


		self.assertIsInstance(handler, GAEHandler)

		with _Monkey(os, getenv=lambda var: var == 'KUBERNETES_SERVICE'):



		class ContainerEngineHandler(logging.StreamHandler):
		"""Handler to formats log messages in good fluentd GKE format.

		"""Python :mod:`logging` handlers for Google Cloud Logging."""

		import logging

Uh oh!

Conversation

waprin commented Nov 3, 2016

Uh oh!

waprin commented Nov 3, 2016

Uh oh!

daspecster left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.