doctor

Health is described with current errors percentage, if the health status turns bad, actions like “refuse service” should be taken, mainly to protect our backend databases.

You must invoke on_* like methods of doctor.checker.HealthTester.metrics(doctor.metrics.Metrics) to record metrics, then HealthTester.is_healthy calculate api call status by thresholds, and HealthTester.test based the flowing policy to decide whether the current request can be passed.

Install

pip install git+https://github.com/eleme/doctor.git

Policy

Current detail policy to test health description:

• if current api is heavily under errors, disallow it to pass the test(), and further incoming requests should be refused ( in at least MIN_RECOVERY_TIME).
• if current api has recoveried from bad health, allow it to pass the test() gradually (via random.random() with priority).

Current errors threholds:

• Errors contains system errors and gevent timeouts.
• Threholds are percentages: errors / requests.
• Errors threholds are checked only if the current requests count is greater than THRESHOLD_REQUEST.

Health check interval:

• Calculated by METRICS_GRANULARITY * METRICS_ROLLINGSIZE, in seconds.

Settings

MIN_RECOVERY_TIME min recovery time (in seconds)
MAX_RECOVERY_TIME max recovery time (in seconds)
THRESHOLD_REQUEST min requests to trigger a health check. (per INTERVAL) # noqa
THRESHOLD_TIMEOUT gevent timeout count threshold (per INTERVAL)
THRESHOLD_SYS_EXC sys_exc count threshold (per INTERVAL)
THRESHOLD_UNKWN_EXC unkwn_exc count threshold (per INTERVAL)

Examples

# callbacks, take *doctor.checker.APIHealthTestCtx* as argumentsdefon_api_health_locked(result):
passdefon_api_health_unlocked(result):
passdefon_api_health_tested(result):
passdefon_api_health_tested_bad(result):
passdefon_api_health_tested_ok(result):
pass# you can custom the settings, see doctor/configs.pyconfigs=Configs()
# callbacks order matters.tester=HealthTester(
configs,
on_api_health_locked,
on_api_health_unlocked,
on_api_health_tested,
on_api_health_tested_bad,
on_api_health_tested_ok,
)
defapi_decorator(func):
@functools.wraps(func)def_wrapper(service, *args, **kwargs):
service_name, func_name=service.name, func.__name__ifnottester.test(service_name, func_name):
print('Oh! No!!!')
returnresult=Nonetry:
result=func(service, *args, **kwargs)
exceptUserError:
tester.metrics.on_api_called_user_exc(service_name, func_name)
exceptTimeoutError:
tester.metrics.on_api_called_timeout(service_name, func_name)
exceptSysError:
tester.metrics.on_api_called_sys_exc(service_name, func_name)
exceptException:
tester.metrics.on_api_called_unkwn_exc(service_name, func_name)
else:
tester.metrics.on_api_called_ok(service_name, func_name)
finally:
tester.metrics.on_api_called(service_name, func_name)
returnresultreturn_wrapper@api_decoratordefapi(service):
client.connect(service.addr)

Ports

• Go

Authors

• @Damnever
• @xiangyu.wang
• @hit9