Skip to content
This repository was archived by the owner on Aug 27, 2026. It is now read-only.
/orcaPublic archive

add resources limit for proper scaling#128

Merged
mag009 merged 8 commits into from
Sep 26, 2018
Merged

add resources limit for proper scaling #128

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
26bde22
add resources limit for proper scaling
mag009 Sep 11, 2018
33f6327
fix comment
mag009 Sep 24, 2018
0d750f2
remove usage of node pool
mag009 Sep 24, 2018
09788ce
set rollingUpdate maxUnavailable to 25%
mag009 Sep 24, 2018
6c812e8
lower cpu request to maximized the utilization of vm
mag009 Sep 24, 2018
1b775cf
adding podAffinity to preferred a different zone
mag009 Sep 24, 2018
b9b603a
add AntiAffinity to have a single pod per hostname
mag009 Sep 24, 2018
1bb3cd2
change to AntiAffinity for failure-domain
mag009 Sep 25, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 39 additions & 8 deletions deployment/kube/prod/frontend.yaml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -10,22 +10,47 @@ metadata:
app: imageserver
spec:
replicas: 3
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: imageserver
tier: frontend
spec:
affinity:

@scjodyscjodySep 18, 2018

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we still need antiAffinity to prevent two pods from ending up on the same node? Or are you counting on resource limits to do that? (Smaller commits, and explaining your reasoning in the commit comment would help here...)

@mag009mag009Sep 18, 2018

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

counting on the resource limit to do that.. and if we do switch for larger instances than we won't care if they spin-up to the same server.

@scjodyscjodySep 18, 2018

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I designed this initially I couldn't find a way to set the resource requests such that one and only one imageserver process could occupy a node, but also allow Kubernetes internal pods to occupy that node.

Is there a way to do it now? This will be an issue if we want to have imageservers in the default node pool, and I think we do.

@mag009mag009Sep 18, 2018

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its the case now that's why i'm using a new pool like we have for redis. This way kube-system won't be allow to run there so we won't affect our critical pods.

The only way i saw it done was with toleration + taint

I don't see why we would want imageservers to run on the default pool. Any reason?

@scjodyscjodySep 19, 2018

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explained my concerns about adding a new node pool briefly in the last paragraph here: #128 (review)

You don't need tolerations and taints to prevent two pods from occupying the same node. That's what the podAntiAffinity statement you're removing was doing, and it was working.

@mag009mag009Sep 20, 2018

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • If I use pod podAntiAffinity to prevent imageserver on kube-system. Will end-up with the same having dedicated machine for kube-system but sharing the same pool... I don't mind doing. Just simpler to use a pool "backend" and put everything else related in there.

  • If I set an podAntiAffinity for imageserver we must make sure to apply it on every app this is where I think a pool make sense.

  • Another situation is when it scale down it evict the kube-system service that is on the node and it restart else where in case of heapster we lose 5 min of metrics. Not a big deal but we might end up with lots of gap in our graph

@scjodyscjodySep 24, 2018

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just suggesting using podAntiAffinity to prevent two imageserver pods from occupying the same node like we do now.

I still don't understand why having Kubernetes internal pods on the same nodes as the imageserver pods is a big deal. They don't use significant amounts of resources, do they? I do understand your concern about losing metrics, but I think it's worth trying anyway. I'm surprised Kubernetes isn't designed to scale down by terminating other nodes rather than these, but it sounds like something we have to live with.

@mag009mag009Sep 24, 2018

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed it back the way it was to use the default-pool and re-added the AntiAffinity to prevent two imageserver on the same host.

  • I've re-ran a stress test and no issue with heapster.

affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- us-central1-a
- us-central1-b
- us-central1-c
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- imageserver
topologyKey: "kubernetes.io/hostname"
- key: app
operator: In
values:
- imageserver
topologyKey: failure-domain.beta.kubernetes.io/zone
weight: 1
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- imageserver
topologyKey: kubernetes.io/hostname
containers:
- name: imageserver-app
image: quay.io/plotly/image-exporter:master
Expand All@@ -40,6 +65,12 @@ spec:
# starting the pod. This is useful when debugging, but should be turned
# off in production.
imagePullPolicy: Always
resources:
limits:
memory: 2396Mi
Comment thread
mag009 marked this conversation as resolved.
requests:
cpu: 100m
memory: 1Gi
ports:
- name: http-server
containerPort: 9091
Expand Down
6 changes: 3 additions & 3 deletions deployment/kube/prod/hpa.yaml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -12,7 +12,7 @@ spec:
kind: Deployment
name: imageserver
# Set this to 3x "min-nodes":
minReplicas: 12
minReplicas: 3
# Set this to 3x "max-nodes":
maxReplicas: 12
targetCPUUtilizationPercentage: 50
maxReplicas: 18
targetCPUUtilizationPercentage: 30
9 changes: 9 additions & 0 deletions deployment/kube/prod/pdb.yaml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: imageserver-pdb
spec:
minAvailable: 2
selector:
matchLabels:
app: imageserver
47 changes: 40 additions & 7 deletions deployment/kube/stage/frontend.yaml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -5,27 +5,54 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
name: imageserver
labels:
app: imageserver
spec:
replicas: 3
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
labels:
app: imageserver
tier: frontend
spec:
affinity:
podAntiAffinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
nodeSelectorTerms:
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- us-central1-a
- us-central1-b
- us-central1-c
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchExpressions:
- key: "app"
operator: In
values:
- imageserver
topologyKey: "kubernetes.io/hostname"
- key: app
operator: In
values:
- imageserver
topologyKey: failure-domain.beta.kubernetes.io/zone
weight: 1
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app
operator: In
values:
- imageserver
topologyKey: kubernetes.io/hostname
containers:
- name: imageserver-app
image: quay.io/plotly/image-exporter:master
Expand All@@ -43,6 +70,12 @@ spec:
ports:
- name: http-server
containerPort: 9091
resources:
limits:
memory: 2396Mi
requests:
cpu: 100m
memory: 1Gi
volumeMounts:
- mountPath: "/usr/share/fonts/user"
name: plotly-cloud-licensed-fonts
Expand Down
4 changes: 2 additions & 2 deletions deployment/kube/stage/hpa.yaml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -14,5 +14,5 @@ spec:
# Set this to 3x "min-nodes":
minReplicas: 3
# Set this to 3x "max-nodes":
maxReplicas: 3
targetCPUUtilizationPercentage: 50
maxReplicas: 6

@scjodyscjodySep 13, 2018

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either the comment needs to be updated, or something else...

@mag009mag009Sep 13, 2018

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noted!

targetCPUUtilizationPercentage: 30
9 changes: 9 additions & 0 deletions deployment/kube/stage/pdb.yaml
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
name: imageserver-pdb
spec:
minAvailable: 2
Comment thread
mag009 marked this conversation as resolved.
selector:
matchLabels:
app: imageserver
2 changes: 1 addition & 1 deletion deployment/run_server
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -26,7 +26,7 @@ fi
pkill Xvfb
pkill node

xvfb-run --auto-servernum --server-args '-screen 0 640x480x24' ./bin/orca.js serve --request-limit=1000 --safe-mode $PLOTLYJS_ARG $@ 1>/proc/1/fd/1 2>/proc/1/fd/2 &
xvfb-run --auto-servernum --server-args '-screen 0 640x480x24' ./bin/orca.js serve --safe-mode $PLOTLYJS_ARG $@ 1>/proc/1/fd/1 2>/proc/1/fd/2 &
echo \$! > \$PIDFILE

EOF
Expand Down
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
add resources limit for proper scaling by mag009 · Pull Request #128 · plotly/orca · GitHub
Skip to content

This repository was archived by the owner on Aug 27, 2026. It is now read-only.

/orcaPublic archive
add resources limit for proper scaling#128
 Merged

mag009
merged 8 commits into

from

Sep 26, 2018
 Merged


add resources limit for proper scaling
#128
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


8 commits


Select commit
Hold shift + click to select a range
26bde22

add resources limit for proper scaling


mag009 Sep 11, 2018
33f6327

fix comment


mag009 Sep 24, 2018
0d750f2

remove usage of node pool


mag009 Sep 24, 2018
09788ce

set rollingUpdate maxUnavailable to 25%


mag009 Sep 24, 2018
6c812e8

lower cpu request to maximized the utilization of vm


mag009 Sep 24, 2018
1b775cf

adding podAffinity to preferred a different zone


mag009 Sep 24, 2018
b9b603a

add AntiAffinity to have a single pod per hostname


mag009 Sep 24, 2018
1bb3cd2

change to AntiAffinity for failure-domain


mag009 Sep 25, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


47 changes: 39 additions & 8 deletions
deployment/kube/prod/frontend.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -10,22 +10,47 @@ metadata:
 app: imageserver
spec:
 replicas: 3
 strategy: 
 rollingUpdate: 
 maxSurge: 100% 
 maxUnavailable: 25%
 type: RollingUpdate 
 template:
 metadata:
 labels:
 app: imageserver
 tier: frontend
 spec:
 affinity:
@scjodyscjodySep 18, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we still need antiAffinity to prevent two pods from ending up on the same node? Or are you counting on resource limits to do that? (Smaller commits, and explaining your reasoning in the commit comment would help here...)
@mag009mag009Sep 18, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

counting on the resource limit to do that.. and if we do switch for larger instances than we won't care if they spin-up to the same server.
@scjodyscjodySep 18, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I designed this initially I couldn't find a way to set the resource requests such that one and only one imageserver process could occupy a node, but also allow Kubernetes internal pods to occupy that node.
Is there a way to do it now? This will be an issue if we want to have imageservers in the default node pool, and I think we do.
@mag009mag009Sep 18, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its the case now that's why i'm using a new pool like we have for redis. This way kube-system won't be allow to run there so we won't affect our critical pods.
The only way i saw it done was with toleration + taint
I don't see why we would want imageservers to run on the default pool. Any reason?
@scjodyscjodySep 19, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explained my concerns about adding a new node pool briefly in the last paragraph here: #128 (review)
You don't need tolerations and taints to prevent two pods from occupying the same node. That's what the podAntiAffinity statement you're removing was doing, and it was working.
@mag009mag009Sep 20, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • If I use pod podAntiAffinity to prevent imageserver on kube-system. Will end-up with the same having dedicated machine for kube-system but sharing the same pool... I don't mind doing. Just simpler to use a pool "backend" and put everything else related in there.
  • If I set an podAntiAffinity for imageserver we must make sure to apply it on every app this is where I think a pool make sense.
  • Another situation is when it scale down it evict the kube-system service that is on the node and it restart else where in case of heapster we lose 5 min of metrics. Not a big deal but we might end up with lots of gap in our graph
@scjodyscjodySep 24, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just suggesting using podAntiAffinity to prevent two imageserver pods from occupying the same node like we do now.
I still don't understand why having Kubernetes internal pods on the same nodes as the imageserver pods is a big deal. They don't use significant amounts of resources, do they? I do understand your concern about losing metrics, but I think it's worth trying anyway. I'm surprised Kubernetes isn't designed to scale down by terminating other nodes rather than these, but it sounds like something we have to live with.
@mag009mag009Sep 24, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed it back the way it was to use the default-pool and re-added the AntiAffinity to prevent two imageserver on the same host.
  • I've re-ran a stress test and no issue with heapster.
 affinity: 
 nodeAffinity: 
 requiredDuringSchedulingIgnoredDuringExecution: 
 nodeSelectorTerms: 
 - matchExpressions: 
 - key: failure-domain.beta.kubernetes.io/zone 
 operator: In 
 values: 
 - us-central1-a 
 - us-central1-b 
 - us-central1-c
 podAntiAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 preferredDuringSchedulingIgnoredDuringExecution:
 - podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: "app"
 operator: In
 values:
 - imageserver
 topologyKey: "kubernetes.io/hostname"
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: failure-domain.beta.kubernetes.io/zone
 weight: 1
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: kubernetes.io/hostname
 containers:
 - name: imageserver-app
 image: quay.io/plotly/image-exporter:master
Expand All@@ -40,6 +65,12 @@ spec:
 # starting the pod. This is useful when debugging, but should be turned
 # off in production.
 imagePullPolicy: Always
 resources: 
 limits: 
 memory: 2396Mi 
Comment thread
mag009 marked this conversation as resolved.

 requests: 
 cpu: 100m
 memory: 1Gi
 ports:
 - name: http-server
 containerPort: 9091
Expand Down

6 changes: 3 additions & 3 deletions
deployment/kube/prod/hpa.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -12,7 +12,7 @@ spec:
 kind: Deployment
 name: imageserver
 # Set this to 3x "min-nodes":
 minReplicas: 12
 minReplicas: 3
 # Set this to 3x "max-nodes":
 maxReplicas: 12
 targetCPUUtilizationPercentage: 50
 maxReplicas: 18
 targetCPUUtilizationPercentage: 30

9 changes: 9 additions & 0 deletions
deployment/kube/prod/pdb.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
 name: imageserver-pdb
spec:
 minAvailable: 2
 selector:
 matchLabels:
 app: imageserver

47 changes: 40 additions & 7 deletions
deployment/kube/stage/frontend.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -5,27 +5,54 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 annotations:
 cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
 name: imageserver
 labels:
 app: imageserver
spec:
 replicas: 3
 strategy:
 rollingUpdate:
 maxSurge: 100%
 maxUnavailable: 25%
 type: RollingUpdate
 template:
 metadata:
 labels:
 app: imageserver
 tier: frontend
 spec:
 affinity:
podAntiAffinity:
nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 nodeSelectorTerms:
 - matchExpressions:
 - key: failure-domain.beta.kubernetes.io/zone
 operator: In
 values:
 - us-central1-a
 - us-central1-b
 - us-central1-c
 podAntiAffinity:
 preferredDuringSchedulingIgnoredDuringExecution:
 - podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: "app"
 operator: In
 values:
 - imageserver
 topologyKey: "kubernetes.io/hostname"
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: failure-domain.beta.kubernetes.io/zone
 weight: 1
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: kubernetes.io/hostname
 containers:
 - name: imageserver-app
 image: quay.io/plotly/image-exporter:master
Expand All@@ -43,6 +70,12 @@ spec:
 ports:
 - name: http-server
 containerPort: 9091
 resources:
 limits:
 memory: 2396Mi
 requests:
 cpu: 100m
 memory: 1Gi
 volumeMounts:
 - mountPath: "/usr/share/fonts/user"
 name: plotly-cloud-licensed-fonts
Expand Down

4 changes: 2 additions & 2 deletions
deployment/kube/stage/hpa.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,5 +14,5 @@ spec:
 # Set this to 3x "min-nodes":
 minReplicas: 3
 # Set this to 3x "max-nodes":
 maxReplicas: 3
 targetCPUUtilizationPercentage: 50
 maxReplicas: 6
@scjodyscjodySep 13, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either the comment needs to be updated, or something else...
@mag009mag009Sep 13, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noted!
 targetCPUUtilizationPercentage: 30

9 changes: 9 additions & 0 deletions
deployment/kube/stage/pdb.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
 name: imageserver-pdb
spec:
 minAvailable: 2
Comment thread
mag009 marked this conversation as resolved.

 selector:
 matchLabels:
 app: imageserver

2 changes: 1 addition & 1 deletion
deployment/run_server

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -26,7 +26,7 @@ fi
pkill Xvfb
pkill node

xvfb-run --auto-servernum --server-args '-screen 0 640x480x24' ./bin/orca.js serve --request-limit=1000 --safe-mode $PLOTLYJS_ARG $@ 1>/proc/1/fd/1 2>/proc/1/fd/2 &
xvfb-run --auto-servernum --server-args '-screen 0 640x480x24' ./bin/orca.js serve --safe-mode $PLOTLYJS_ARG $@ 1>/proc/1/fd/1 2>/proc/1/fd/2 &
echo \$! > \$PIDFILE

EOF
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
add resources limit for proper scaling by mag009 · Pull Request #128 · plotly/orca · GitHub
Skip to content

This repository was archived by the owner on Aug 27, 2026. It is now read-only.

/orcaPublic archive
add resources limit for proper scaling#128
 Merged

mag009
merged 8 commits into

from

Sep 26, 2018
 Merged


add resources limit for proper scaling
#128
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


8 commits


Select commit
Hold shift + click to select a range
26bde22

add resources limit for proper scaling


mag009 Sep 11, 2018
33f6327

fix comment


mag009 Sep 24, 2018
0d750f2

remove usage of node pool


mag009 Sep 24, 2018
09788ce

set rollingUpdate maxUnavailable to 25%


mag009 Sep 24, 2018
6c812e8

lower cpu request to maximized the utilization of vm


mag009 Sep 24, 2018
1b775cf

adding podAffinity to preferred a different zone


mag009 Sep 24, 2018
b9b603a

add AntiAffinity to have a single pod per hostname


mag009 Sep 24, 2018
1bb3cd2

change to AntiAffinity for failure-domain


mag009 Sep 25, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


47 changes: 39 additions & 8 deletions
deployment/kube/prod/frontend.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -10,22 +10,47 @@ metadata:
 app: imageserver
spec:
 replicas: 3
 strategy: 
 rollingUpdate: 
 maxSurge: 100% 
 maxUnavailable: 25%
 type: RollingUpdate 
 template:
 metadata:
 labels:
 app: imageserver
 tier: frontend
 spec:
 affinity:
@scjodyscjodySep 18, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we still need antiAffinity to prevent two pods from ending up on the same node? Or are you counting on resource limits to do that? (Smaller commits, and explaining your reasoning in the commit comment would help here...)
@mag009mag009Sep 18, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

counting on the resource limit to do that.. and if we do switch for larger instances than we won't care if they spin-up to the same server.
@scjodyscjodySep 18, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I designed this initially I couldn't find a way to set the resource requests such that one and only one imageserver process could occupy a node, but also allow Kubernetes internal pods to occupy that node.
Is there a way to do it now? This will be an issue if we want to have imageservers in the default node pool, and I think we do.
@mag009mag009Sep 18, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its the case now that's why i'm using a new pool like we have for redis. This way kube-system won't be allow to run there so we won't affect our critical pods.
The only way i saw it done was with toleration + taint
I don't see why we would want imageservers to run on the default pool. Any reason?
@scjodyscjodySep 19, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explained my concerns about adding a new node pool briefly in the last paragraph here: #128 (review)
You don't need tolerations and taints to prevent two pods from occupying the same node. That's what the podAntiAffinity statement you're removing was doing, and it was working.
@mag009mag009Sep 20, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • If I use pod podAntiAffinity to prevent imageserver on kube-system. Will end-up with the same having dedicated machine for kube-system but sharing the same pool... I don't mind doing. Just simpler to use a pool "backend" and put everything else related in there.
  • If I set an podAntiAffinity for imageserver we must make sure to apply it on every app this is where I think a pool make sense.
  • Another situation is when it scale down it evict the kube-system service that is on the node and it restart else where in case of heapster we lose 5 min of metrics. Not a big deal but we might end up with lots of gap in our graph
@scjodyscjodySep 24, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just suggesting using podAntiAffinity to prevent two imageserver pods from occupying the same node like we do now.
I still don't understand why having Kubernetes internal pods on the same nodes as the imageserver pods is a big deal. They don't use significant amounts of resources, do they? I do understand your concern about losing metrics, but I think it's worth trying anyway. I'm surprised Kubernetes isn't designed to scale down by terminating other nodes rather than these, but it sounds like something we have to live with.
@mag009mag009Sep 24, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed it back the way it was to use the default-pool and re-added the AntiAffinity to prevent two imageserver on the same host.
  • I've re-ran a stress test and no issue with heapster.
 affinity: 
 nodeAffinity: 
 requiredDuringSchedulingIgnoredDuringExecution: 
 nodeSelectorTerms: 
 - matchExpressions: 
 - key: failure-domain.beta.kubernetes.io/zone 
 operator: In 
 values: 
 - us-central1-a 
 - us-central1-b 
 - us-central1-c
 podAntiAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 preferredDuringSchedulingIgnoredDuringExecution:
 - podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: "app"
 operator: In
 values:
 - imageserver
 topologyKey: "kubernetes.io/hostname"
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: failure-domain.beta.kubernetes.io/zone
 weight: 1
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: kubernetes.io/hostname
 containers:
 - name: imageserver-app
 image: quay.io/plotly/image-exporter:master
Expand All@@ -40,6 +65,12 @@ spec:
 # starting the pod. This is useful when debugging, but should be turned
 # off in production.
 imagePullPolicy: Always
 resources: 
 limits: 
 memory: 2396Mi 
Comment thread
mag009 marked this conversation as resolved.

 requests: 
 cpu: 100m
 memory: 1Gi
 ports:
 - name: http-server
 containerPort: 9091
Expand Down

6 changes: 3 additions & 3 deletions
deployment/kube/prod/hpa.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -12,7 +12,7 @@ spec:
 kind: Deployment
 name: imageserver
 # Set this to 3x "min-nodes":
 minReplicas: 12
 minReplicas: 3
 # Set this to 3x "max-nodes":
 maxReplicas: 12
 targetCPUUtilizationPercentage: 50
 maxReplicas: 18
 targetCPUUtilizationPercentage: 30

9 changes: 9 additions & 0 deletions
deployment/kube/prod/pdb.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
 name: imageserver-pdb
spec:
 minAvailable: 2
 selector:
 matchLabels:
 app: imageserver

47 changes: 40 additions & 7 deletions
deployment/kube/stage/frontend.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -5,27 +5,54 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 annotations:
 cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
 name: imageserver
 labels:
 app: imageserver
spec:
 replicas: 3
 strategy:
 rollingUpdate:
 maxSurge: 100%
 maxUnavailable: 25%
 type: RollingUpdate
 template:
 metadata:
 labels:
 app: imageserver
 tier: frontend
 spec:
 affinity:
podAntiAffinity:
nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 nodeSelectorTerms:
 - matchExpressions:
 - key: failure-domain.beta.kubernetes.io/zone
 operator: In
 values:
 - us-central1-a
 - us-central1-b
 - us-central1-c
 podAntiAffinity:
 preferredDuringSchedulingIgnoredDuringExecution:
 - podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: "app"
 operator: In
 values:
 - imageserver
 topologyKey: "kubernetes.io/hostname"
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: failure-domain.beta.kubernetes.io/zone
 weight: 1
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: kubernetes.io/hostname
 containers:
 - name: imageserver-app
 image: quay.io/plotly/image-exporter:master
Expand All@@ -43,6 +70,12 @@ spec:
 ports:
 - name: http-server
 containerPort: 9091
 resources:
 limits:
 memory: 2396Mi
 requests:
 cpu: 100m
 memory: 1Gi
 volumeMounts:
 - mountPath: "/usr/share/fonts/user"
 name: plotly-cloud-licensed-fonts
Expand Down

4 changes: 2 additions & 2 deletions
deployment/kube/stage/hpa.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,5 +14,5 @@ spec:
 # Set this to 3x "min-nodes":
 minReplicas: 3
 # Set this to 3x "max-nodes":
 maxReplicas: 3
 targetCPUUtilizationPercentage: 50
 maxReplicas: 6
@scjodyscjodySep 13, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either the comment needs to be updated, or something else...
@mag009mag009Sep 13, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noted!
 targetCPUUtilizationPercentage: 30

9 changes: 9 additions & 0 deletions
deployment/kube/stage/pdb.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
 name: imageserver-pdb
spec:
 minAvailable: 2
Comment thread
mag009 marked this conversation as resolved.

 selector:
 matchLabels:
 app: imageserver

2 changes: 1 addition & 1 deletion
deployment/run_server

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -26,7 +26,7 @@ fi
pkill Xvfb
pkill node

xvfb-run --auto-servernum --server-args '-screen 0 640x480x24' ./bin/orca.js serve --request-limit=1000 --safe-mode $PLOTLYJS_ARG $@ 1>/proc/1/fd/1 2>/proc/1/fd/2 &
xvfb-run --auto-servernum --server-args '-screen 0 640x480x24' ./bin/orca.js serve --safe-mode $PLOTLYJS_ARG $@ 1>/proc/1/fd/1 2>/proc/1/fd/2 &
echo \$! > \$PIDFILE

EOF
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
add resources limit for proper scaling by mag009 · Pull Request #128 · plotly/orca · GitHub
Skip to content

This repository was archived by the owner on Aug 27, 2026. It is now read-only.

/orcaPublic archive
add resources limit for proper scaling#128
 Merged

mag009
merged 8 commits into

from

Sep 26, 2018
 Merged


add resources limit for proper scaling
#128
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


8 commits


Select commit
Hold shift + click to select a range
26bde22

add resources limit for proper scaling


mag009 Sep 11, 2018
33f6327

fix comment


mag009 Sep 24, 2018
0d750f2

remove usage of node pool


mag009 Sep 24, 2018
09788ce

set rollingUpdate maxUnavailable to 25%


mag009 Sep 24, 2018
6c812e8

lower cpu request to maximized the utilization of vm


mag009 Sep 24, 2018
1b775cf

adding podAffinity to preferred a different zone


mag009 Sep 24, 2018
b9b603a

add AntiAffinity to have a single pod per hostname


mag009 Sep 24, 2018
1bb3cd2

change to AntiAffinity for failure-domain


mag009 Sep 25, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


47 changes: 39 additions & 8 deletions
deployment/kube/prod/frontend.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -10,22 +10,47 @@ metadata:
 app: imageserver
spec:
 replicas: 3
 strategy: 
 rollingUpdate: 
 maxSurge: 100% 
 maxUnavailable: 25%
 type: RollingUpdate 
 template:
 metadata:
 labels:
 app: imageserver
 tier: frontend
 spec:
 affinity:
@scjodyscjodySep 18, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we still need antiAffinity to prevent two pods from ending up on the same node? Or are you counting on resource limits to do that? (Smaller commits, and explaining your reasoning in the commit comment would help here...)
@mag009mag009Sep 18, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

counting on the resource limit to do that.. and if we do switch for larger instances than we won't care if they spin-up to the same server.
@scjodyscjodySep 18, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I designed this initially I couldn't find a way to set the resource requests such that one and only one imageserver process could occupy a node, but also allow Kubernetes internal pods to occupy that node.
Is there a way to do it now? This will be an issue if we want to have imageservers in the default node pool, and I think we do.
@mag009mag009Sep 18, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its the case now that's why i'm using a new pool like we have for redis. This way kube-system won't be allow to run there so we won't affect our critical pods.
The only way i saw it done was with toleration + taint
I don't see why we would want imageservers to run on the default pool. Any reason?
@scjodyscjodySep 19, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explained my concerns about adding a new node pool briefly in the last paragraph here: #128 (review)
You don't need tolerations and taints to prevent two pods from occupying the same node. That's what the podAntiAffinity statement you're removing was doing, and it was working.
@mag009mag009Sep 20, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • If I use pod podAntiAffinity to prevent imageserver on kube-system. Will end-up with the same having dedicated machine for kube-system but sharing the same pool... I don't mind doing. Just simpler to use a pool "backend" and put everything else related in there.
  • If I set an podAntiAffinity for imageserver we must make sure to apply it on every app this is where I think a pool make sense.
  • Another situation is when it scale down it evict the kube-system service that is on the node and it restart else where in case of heapster we lose 5 min of metrics. Not a big deal but we might end up with lots of gap in our graph
@scjodyscjodySep 24, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just suggesting using podAntiAffinity to prevent two imageserver pods from occupying the same node like we do now.
I still don't understand why having Kubernetes internal pods on the same nodes as the imageserver pods is a big deal. They don't use significant amounts of resources, do they? I do understand your concern about losing metrics, but I think it's worth trying anyway. I'm surprised Kubernetes isn't designed to scale down by terminating other nodes rather than these, but it sounds like something we have to live with.
@mag009mag009Sep 24, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed it back the way it was to use the default-pool and re-added the AntiAffinity to prevent two imageserver on the same host.
  • I've re-ran a stress test and no issue with heapster.
 affinity: 
 nodeAffinity: 
 requiredDuringSchedulingIgnoredDuringExecution: 
 nodeSelectorTerms: 
 - matchExpressions: 
 - key: failure-domain.beta.kubernetes.io/zone 
 operator: In 
 values: 
 - us-central1-a 
 - us-central1-b 
 - us-central1-c
 podAntiAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 preferredDuringSchedulingIgnoredDuringExecution:
 - podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: "app"
 operator: In
 values:
 - imageserver
 topologyKey: "kubernetes.io/hostname"
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: failure-domain.beta.kubernetes.io/zone
 weight: 1
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: kubernetes.io/hostname
 containers:
 - name: imageserver-app
 image: quay.io/plotly/image-exporter:master
Expand All@@ -40,6 +65,12 @@ spec:
 # starting the pod. This is useful when debugging, but should be turned
 # off in production.
 imagePullPolicy: Always
 resources: 
 limits: 
 memory: 2396Mi 
Comment thread
mag009 marked this conversation as resolved.

 requests: 
 cpu: 100m
 memory: 1Gi
 ports:
 - name: http-server
 containerPort: 9091
Expand Down

6 changes: 3 additions & 3 deletions
deployment/kube/prod/hpa.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -12,7 +12,7 @@ spec:
 kind: Deployment
 name: imageserver
 # Set this to 3x "min-nodes":
 minReplicas: 12
 minReplicas: 3
 # Set this to 3x "max-nodes":
 maxReplicas: 12
 targetCPUUtilizationPercentage: 50
 maxReplicas: 18
 targetCPUUtilizationPercentage: 30

9 changes: 9 additions & 0 deletions
deployment/kube/prod/pdb.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
 name: imageserver-pdb
spec:
 minAvailable: 2
 selector:
 matchLabels:
 app: imageserver

47 changes: 40 additions & 7 deletions
deployment/kube/stage/frontend.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -5,27 +5,54 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 annotations:
 cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
 name: imageserver
 labels:
 app: imageserver
spec:
 replicas: 3
 strategy:
 rollingUpdate:
 maxSurge: 100%
 maxUnavailable: 25%
 type: RollingUpdate
 template:
 metadata:
 labels:
 app: imageserver
 tier: frontend
 spec:
 affinity:
podAntiAffinity:
nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 nodeSelectorTerms:
 - matchExpressions:
 - key: failure-domain.beta.kubernetes.io/zone
 operator: In
 values:
 - us-central1-a
 - us-central1-b
 - us-central1-c
 podAntiAffinity:
 preferredDuringSchedulingIgnoredDuringExecution:
 - podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: "app"
 operator: In
 values:
 - imageserver
 topologyKey: "kubernetes.io/hostname"
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: failure-domain.beta.kubernetes.io/zone
 weight: 1
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: kubernetes.io/hostname
 containers:
 - name: imageserver-app
 image: quay.io/plotly/image-exporter:master
Expand All@@ -43,6 +70,12 @@ spec:
 ports:
 - name: http-server
 containerPort: 9091
 resources:
 limits:
 memory: 2396Mi
 requests:
 cpu: 100m
 memory: 1Gi
 volumeMounts:
 - mountPath: "/usr/share/fonts/user"
 name: plotly-cloud-licensed-fonts
Expand Down

4 changes: 2 additions & 2 deletions
deployment/kube/stage/hpa.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,5 +14,5 @@ spec:
 # Set this to 3x "min-nodes":
 minReplicas: 3
 # Set this to 3x "max-nodes":
 maxReplicas: 3
 targetCPUUtilizationPercentage: 50
 maxReplicas: 6
@scjodyscjodySep 13, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either the comment needs to be updated, or something else...
@mag009mag009Sep 13, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noted!
 targetCPUUtilizationPercentage: 30

9 changes: 9 additions & 0 deletions
deployment/kube/stage/pdb.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
 name: imageserver-pdb
spec:
 minAvailable: 2
Comment thread
mag009 marked this conversation as resolved.

 selector:
 matchLabels:
 app: imageserver

2 changes: 1 addition & 1 deletion
deployment/run_server

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -26,7 +26,7 @@ fi
pkill Xvfb
pkill node

xvfb-run --auto-servernum --server-args '-screen 0 640x480x24' ./bin/orca.js serve --request-limit=1000 --safe-mode $PLOTLYJS_ARG $@ 1>/proc/1/fd/1 2>/proc/1/fd/2 &
xvfb-run --auto-servernum --server-args '-screen 0 640x480x24' ./bin/orca.js serve --safe-mode $PLOTLYJS_ARG $@ 1>/proc/1/fd/1 2>/proc/1/fd/2 &
echo \$! > \$PIDFILE

EOF
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
add resources limit for proper scaling by mag009 · Pull Request #128 · plotly/orca · GitHub
Skip to content

This repository was archived by the owner on Aug 27, 2026. It is now read-only.

/orcaPublic archive
add resources limit for proper scaling#128
 Merged

mag009
merged 8 commits into

from

Sep 26, 2018
 Merged


add resources limit for proper scaling
#128
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


8 commits


Select commit
Hold shift + click to select a range
26bde22

add resources limit for proper scaling


mag009 Sep 11, 2018
33f6327

fix comment


mag009 Sep 24, 2018
0d750f2

remove usage of node pool


mag009 Sep 24, 2018
09788ce

set rollingUpdate maxUnavailable to 25%


mag009 Sep 24, 2018
6c812e8

lower cpu request to maximized the utilization of vm


mag009 Sep 24, 2018
1b775cf

adding podAffinity to preferred a different zone


mag009 Sep 24, 2018
b9b603a

add AntiAffinity to have a single pod per hostname


mag009 Sep 24, 2018
1bb3cd2

change to AntiAffinity for failure-domain


mag009 Sep 25, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


47 changes: 39 additions & 8 deletions
deployment/kube/prod/frontend.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -10,22 +10,47 @@ metadata:
 app: imageserver
spec:
 replicas: 3
 strategy: 
 rollingUpdate: 
 maxSurge: 100% 
 maxUnavailable: 25%
 type: RollingUpdate 
 template:
 metadata:
 labels:
 app: imageserver
 tier: frontend
 spec:
 affinity:
@scjodyscjodySep 18, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we still need antiAffinity to prevent two pods from ending up on the same node? Or are you counting on resource limits to do that? (Smaller commits, and explaining your reasoning in the commit comment would help here...)
@mag009mag009Sep 18, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

counting on the resource limit to do that.. and if we do switch for larger instances than we won't care if they spin-up to the same server.
@scjodyscjodySep 18, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I designed this initially I couldn't find a way to set the resource requests such that one and only one imageserver process could occupy a node, but also allow Kubernetes internal pods to occupy that node.
Is there a way to do it now? This will be an issue if we want to have imageservers in the default node pool, and I think we do.
@mag009mag009Sep 18, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its the case now that's why i'm using a new pool like we have for redis. This way kube-system won't be allow to run there so we won't affect our critical pods.
The only way i saw it done was with toleration + taint
I don't see why we would want imageservers to run on the default pool. Any reason?
@scjodyscjodySep 19, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explained my concerns about adding a new node pool briefly in the last paragraph here: #128 (review)
You don't need tolerations and taints to prevent two pods from occupying the same node. That's what the podAntiAffinity statement you're removing was doing, and it was working.
@mag009mag009Sep 20, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • If I use pod podAntiAffinity to prevent imageserver on kube-system. Will end-up with the same having dedicated machine for kube-system but sharing the same pool... I don't mind doing. Just simpler to use a pool "backend" and put everything else related in there.
  • If I set an podAntiAffinity for imageserver we must make sure to apply it on every app this is where I think a pool make sense.
  • Another situation is when it scale down it evict the kube-system service that is on the node and it restart else where in case of heapster we lose 5 min of metrics. Not a big deal but we might end up with lots of gap in our graph
@scjodyscjodySep 24, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just suggesting using podAntiAffinity to prevent two imageserver pods from occupying the same node like we do now.
I still don't understand why having Kubernetes internal pods on the same nodes as the imageserver pods is a big deal. They don't use significant amounts of resources, do they? I do understand your concern about losing metrics, but I think it's worth trying anyway. I'm surprised Kubernetes isn't designed to scale down by terminating other nodes rather than these, but it sounds like something we have to live with.
@mag009mag009Sep 24, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed it back the way it was to use the default-pool and re-added the AntiAffinity to prevent two imageserver on the same host.
  • I've re-ran a stress test and no issue with heapster.
 affinity: 
 nodeAffinity: 
 requiredDuringSchedulingIgnoredDuringExecution: 
 nodeSelectorTerms: 
 - matchExpressions: 
 - key: failure-domain.beta.kubernetes.io/zone 
 operator: In 
 values: 
 - us-central1-a 
 - us-central1-b 
 - us-central1-c
 podAntiAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 preferredDuringSchedulingIgnoredDuringExecution:
 - podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: "app"
 operator: In
 values:
 - imageserver
 topologyKey: "kubernetes.io/hostname"
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: failure-domain.beta.kubernetes.io/zone
 weight: 1
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: kubernetes.io/hostname
 containers:
 - name: imageserver-app
 image: quay.io/plotly/image-exporter:master
Expand All@@ -40,6 +65,12 @@ spec:
 # starting the pod. This is useful when debugging, but should be turned
 # off in production.
 imagePullPolicy: Always
 resources: 
 limits: 
 memory: 2396Mi 
Comment thread
mag009 marked this conversation as resolved.

 requests: 
 cpu: 100m
 memory: 1Gi
 ports:
 - name: http-server
 containerPort: 9091
Expand Down

6 changes: 3 additions & 3 deletions
deployment/kube/prod/hpa.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -12,7 +12,7 @@ spec:
 kind: Deployment
 name: imageserver
 # Set this to 3x "min-nodes":
 minReplicas: 12
 minReplicas: 3
 # Set this to 3x "max-nodes":
 maxReplicas: 12
 targetCPUUtilizationPercentage: 50
 maxReplicas: 18
 targetCPUUtilizationPercentage: 30

9 changes: 9 additions & 0 deletions
deployment/kube/prod/pdb.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
 name: imageserver-pdb
spec:
 minAvailable: 2
 selector:
 matchLabels:
 app: imageserver

47 changes: 40 additions & 7 deletions
deployment/kube/stage/frontend.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -5,27 +5,54 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 annotations:
 cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
 name: imageserver
 labels:
 app: imageserver
spec:
 replicas: 3
 strategy:
 rollingUpdate:
 maxSurge: 100%
 maxUnavailable: 25%
 type: RollingUpdate
 template:
 metadata:
 labels:
 app: imageserver
 tier: frontend
 spec:
 affinity:
podAntiAffinity:
nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 nodeSelectorTerms:
 - matchExpressions:
 - key: failure-domain.beta.kubernetes.io/zone
 operator: In
 values:
 - us-central1-a
 - us-central1-b
 - us-central1-c
 podAntiAffinity:
 preferredDuringSchedulingIgnoredDuringExecution:
 - podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: "app"
 operator: In
 values:
 - imageserver
 topologyKey: "kubernetes.io/hostname"
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: failure-domain.beta.kubernetes.io/zone
 weight: 1
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: kubernetes.io/hostname
 containers:
 - name: imageserver-app
 image: quay.io/plotly/image-exporter:master
Expand All@@ -43,6 +70,12 @@ spec:
 ports:
 - name: http-server
 containerPort: 9091
 resources:
 limits:
 memory: 2396Mi
 requests:
 cpu: 100m
 memory: 1Gi
 volumeMounts:
 - mountPath: "/usr/share/fonts/user"
 name: plotly-cloud-licensed-fonts
Expand Down

4 changes: 2 additions & 2 deletions
deployment/kube/stage/hpa.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,5 +14,5 @@ spec:
 # Set this to 3x "min-nodes":
 minReplicas: 3
 # Set this to 3x "max-nodes":
 maxReplicas: 3
 targetCPUUtilizationPercentage: 50
 maxReplicas: 6
@scjodyscjodySep 13, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either the comment needs to be updated, or something else...
@mag009mag009Sep 13, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noted!
 targetCPUUtilizationPercentage: 30

9 changes: 9 additions & 0 deletions
deployment/kube/stage/pdb.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
 name: imageserver-pdb
spec:
 minAvailable: 2
Comment thread
mag009 marked this conversation as resolved.

 selector:
 matchLabels:
 app: imageserver

2 changes: 1 addition & 1 deletion
deployment/run_server

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -26,7 +26,7 @@ fi
pkill Xvfb
pkill node

xvfb-run --auto-servernum --server-args '-screen 0 640x480x24' ./bin/orca.js serve --request-limit=1000 --safe-mode $PLOTLYJS_ARG $@ 1>/proc/1/fd/1 2>/proc/1/fd/2 &
xvfb-run --auto-servernum --server-args '-screen 0 640x480x24' ./bin/orca.js serve --safe-mode $PLOTLYJS_ARG $@ 1>/proc/1/fd/1 2>/proc/1/fd/2 &
echo \$! > \$PIDFILE

EOF
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
add resources limit for proper scaling by mag009 · Pull Request #128 · plotly/orca · GitHub
Skip to content

This repository was archived by the owner on Aug 27, 2026. It is now read-only.

/orcaPublic archive
add resources limit for proper scaling#128
 Merged

mag009
merged 8 commits into

from

Sep 26, 2018
 Merged


add resources limit for proper scaling
#128
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


8 commits


Select commit
Hold shift + click to select a range
26bde22

add resources limit for proper scaling


mag009 Sep 11, 2018
33f6327

fix comment


mag009 Sep 24, 2018
0d750f2

remove usage of node pool


mag009 Sep 24, 2018
09788ce

set rollingUpdate maxUnavailable to 25%


mag009 Sep 24, 2018
6c812e8

lower cpu request to maximized the utilization of vm


mag009 Sep 24, 2018
1b775cf

adding podAffinity to preferred a different zone


mag009 Sep 24, 2018
b9b603a

add AntiAffinity to have a single pod per hostname


mag009 Sep 24, 2018
1bb3cd2

change to AntiAffinity for failure-domain


mag009 Sep 25, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


47 changes: 39 additions & 8 deletions
deployment/kube/prod/frontend.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -10,22 +10,47 @@ metadata:
 app: imageserver
spec:
 replicas: 3
 strategy: 
 rollingUpdate: 
 maxSurge: 100% 
 maxUnavailable: 25%
 type: RollingUpdate 
 template:
 metadata:
 labels:
 app: imageserver
 tier: frontend
 spec:
 affinity:
@scjodyscjodySep 18, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we still need antiAffinity to prevent two pods from ending up on the same node? Or are you counting on resource limits to do that? (Smaller commits, and explaining your reasoning in the commit comment would help here...)
@mag009mag009Sep 18, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

counting on the resource limit to do that.. and if we do switch for larger instances than we won't care if they spin-up to the same server.
@scjodyscjodySep 18, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I designed this initially I couldn't find a way to set the resource requests such that one and only one imageserver process could occupy a node, but also allow Kubernetes internal pods to occupy that node.
Is there a way to do it now? This will be an issue if we want to have imageservers in the default node pool, and I think we do.
@mag009mag009Sep 18, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its the case now that's why i'm using a new pool like we have for redis. This way kube-system won't be allow to run there so we won't affect our critical pods.
The only way i saw it done was with toleration + taint
I don't see why we would want imageservers to run on the default pool. Any reason?
@scjodyscjodySep 19, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explained my concerns about adding a new node pool briefly in the last paragraph here: #128 (review)
You don't need tolerations and taints to prevent two pods from occupying the same node. That's what the podAntiAffinity statement you're removing was doing, and it was working.
@mag009mag009Sep 20, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • If I use pod podAntiAffinity to prevent imageserver on kube-system. Will end-up with the same having dedicated machine for kube-system but sharing the same pool... I don't mind doing. Just simpler to use a pool "backend" and put everything else related in there.
  • If I set an podAntiAffinity for imageserver we must make sure to apply it on every app this is where I think a pool make sense.
  • Another situation is when it scale down it evict the kube-system service that is on the node and it restart else where in case of heapster we lose 5 min of metrics. Not a big deal but we might end up with lots of gap in our graph
@scjodyscjodySep 24, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just suggesting using podAntiAffinity to prevent two imageserver pods from occupying the same node like we do now.
I still don't understand why having Kubernetes internal pods on the same nodes as the imageserver pods is a big deal. They don't use significant amounts of resources, do they? I do understand your concern about losing metrics, but I think it's worth trying anyway. I'm surprised Kubernetes isn't designed to scale down by terminating other nodes rather than these, but it sounds like something we have to live with.
@mag009mag009Sep 24, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed it back the way it was to use the default-pool and re-added the AntiAffinity to prevent two imageserver on the same host.
  • I've re-ran a stress test and no issue with heapster.
 affinity: 
 nodeAffinity: 
 requiredDuringSchedulingIgnoredDuringExecution: 
 nodeSelectorTerms: 
 - matchExpressions: 
 - key: failure-domain.beta.kubernetes.io/zone 
 operator: In 
 values: 
 - us-central1-a 
 - us-central1-b 
 - us-central1-c
 podAntiAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 preferredDuringSchedulingIgnoredDuringExecution:
 - podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: "app"
 operator: In
 values:
 - imageserver
 topologyKey: "kubernetes.io/hostname"
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: failure-domain.beta.kubernetes.io/zone
 weight: 1
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: kubernetes.io/hostname
 containers:
 - name: imageserver-app
 image: quay.io/plotly/image-exporter:master
Expand All@@ -40,6 +65,12 @@ spec:
 # starting the pod. This is useful when debugging, but should be turned
 # off in production.
 imagePullPolicy: Always
 resources: 
 limits: 
 memory: 2396Mi 
Comment thread
mag009 marked this conversation as resolved.

 requests: 
 cpu: 100m
 memory: 1Gi
 ports:
 - name: http-server
 containerPort: 9091
Expand Down

6 changes: 3 additions & 3 deletions
deployment/kube/prod/hpa.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -12,7 +12,7 @@ spec:
 kind: Deployment
 name: imageserver
 # Set this to 3x "min-nodes":
 minReplicas: 12
 minReplicas: 3
 # Set this to 3x "max-nodes":
 maxReplicas: 12
 targetCPUUtilizationPercentage: 50
 maxReplicas: 18
 targetCPUUtilizationPercentage: 30

9 changes: 9 additions & 0 deletions
deployment/kube/prod/pdb.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
 name: imageserver-pdb
spec:
 minAvailable: 2
 selector:
 matchLabels:
 app: imageserver

47 changes: 40 additions & 7 deletions
deployment/kube/stage/frontend.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -5,27 +5,54 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 annotations:
 cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
 name: imageserver
 labels:
 app: imageserver
spec:
 replicas: 3
 strategy:
 rollingUpdate:
 maxSurge: 100%
 maxUnavailable: 25%
 type: RollingUpdate
 template:
 metadata:
 labels:
 app: imageserver
 tier: frontend
 spec:
 affinity:
podAntiAffinity:
nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 nodeSelectorTerms:
 - matchExpressions:
 - key: failure-domain.beta.kubernetes.io/zone
 operator: In
 values:
 - us-central1-a
 - us-central1-b
 - us-central1-c
 podAntiAffinity:
 preferredDuringSchedulingIgnoredDuringExecution:
 - podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: "app"
 operator: In
 values:
 - imageserver
 topologyKey: "kubernetes.io/hostname"
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: failure-domain.beta.kubernetes.io/zone
 weight: 1
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: kubernetes.io/hostname
 containers:
 - name: imageserver-app
 image: quay.io/plotly/image-exporter:master
Expand All@@ -43,6 +70,12 @@ spec:
 ports:
 - name: http-server
 containerPort: 9091
 resources:
 limits:
 memory: 2396Mi
 requests:
 cpu: 100m
 memory: 1Gi
 volumeMounts:
 - mountPath: "/usr/share/fonts/user"
 name: plotly-cloud-licensed-fonts
Expand Down

4 changes: 2 additions & 2 deletions
deployment/kube/stage/hpa.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,5 +14,5 @@ spec:
 # Set this to 3x "min-nodes":
 minReplicas: 3
 # Set this to 3x "max-nodes":
 maxReplicas: 3
 targetCPUUtilizationPercentage: 50
 maxReplicas: 6
@scjodyscjodySep 13, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either the comment needs to be updated, or something else...
@mag009mag009Sep 13, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noted!
 targetCPUUtilizationPercentage: 30

9 changes: 9 additions & 0 deletions
deployment/kube/stage/pdb.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
 name: imageserver-pdb
spec:
 minAvailable: 2
Comment thread
mag009 marked this conversation as resolved.

 selector:
 matchLabels:
 app: imageserver

2 changes: 1 addition & 1 deletion
deployment/run_server

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -26,7 +26,7 @@ fi
pkill Xvfb
pkill node

xvfb-run --auto-servernum --server-args '-screen 0 640x480x24' ./bin/orca.js serve --request-limit=1000 --safe-mode $PLOTLYJS_ARG $@ 1>/proc/1/fd/1 2>/proc/1/fd/2 &
xvfb-run --auto-servernum --server-args '-screen 0 640x480x24' ./bin/orca.js serve --safe-mode $PLOTLYJS_ARG $@ 1>/proc/1/fd/1 2>/proc/1/fd/2 &
echo \$! > \$PIDFILE

EOF
Expand Down

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
})();
add resources limit for proper scaling by mag009 · Pull Request #128 · plotly/orca · GitHub
Skip to content

This repository was archived by the owner on Aug 27, 2026. It is now read-only.

/orcaPublic archive
add resources limit for proper scaling#128
 Merged

mag009
merged 8 commits into

from

Sep 26, 2018
 Merged


add resources limit for proper scaling
#128
Show file tree
Hide file tree
 Changes from all commits
Commits

Show all changes


8 commits


Select commit
Hold shift + click to select a range
26bde22

add resources limit for proper scaling


mag009 Sep 11, 2018
33f6327

fix comment


mag009 Sep 24, 2018
0d750f2

remove usage of node pool


mag009 Sep 24, 2018
09788ce

set rollingUpdate maxUnavailable to 25%


mag009 Sep 24, 2018
6c812e8

lower cpu request to maximized the utilization of vm


mag009 Sep 24, 2018
1b775cf

adding podAffinity to preferred a different zone


mag009 Sep 24, 2018
b9b603a

add AntiAffinity to have a single pod per hostname


mag009 Sep 24, 2018
1bb3cd2

change to AntiAffinity for failure-domain


mag009 Sep 25, 2018

File filter

Filter by extension
Filter by extension
Conversations

Failed to load comments. 
Loading
Jump to

Jump to file


Failed to load files. 
Loading
 Diff view

 Diff view


47 changes: 39 additions & 8 deletions
deployment/kube/prod/frontend.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -10,22 +10,47 @@ metadata:
 app: imageserver
spec:
 replicas: 3
 strategy: 
 rollingUpdate: 
 maxSurge: 100% 
 maxUnavailable: 25%
 type: RollingUpdate 
 template:
 metadata:
 labels:
 app: imageserver
 tier: frontend
 spec:
 affinity:
@scjodyscjodySep 18, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't we still need antiAffinity to prevent two pods from ending up on the same node? Or are you counting on resource limits to do that? (Smaller commits, and explaining your reasoning in the commit comment would help here...)
@mag009mag009Sep 18, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

counting on the resource limit to do that.. and if we do switch for larger instances than we won't care if they spin-up to the same server.
@scjodyscjodySep 18, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I designed this initially I couldn't find a way to set the resource requests such that one and only one imageserver process could occupy a node, but also allow Kubernetes internal pods to occupy that node.
Is there a way to do it now? This will be an issue if we want to have imageservers in the default node pool, and I think we do.
@mag009mag009Sep 18, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

its the case now that's why i'm using a new pool like we have for redis. This way kube-system won't be allow to run there so we won't affect our critical pods.
The only way i saw it done was with toleration + taint
I don't see why we would want imageservers to run on the default pool. Any reason?
@scjodyscjodySep 19, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explained my concerns about adding a new node pool briefly in the last paragraph here: #128 (review)
You don't need tolerations and taints to prevent two pods from occupying the same node. That's what the podAntiAffinity statement you're removing was doing, and it was working.
@mag009mag009Sep 20, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • If I use pod podAntiAffinity to prevent imageserver on kube-system. Will end-up with the same having dedicated machine for kube-system but sharing the same pool... I don't mind doing. Just simpler to use a pool "backend" and put everything else related in there.
  • If I set an podAntiAffinity for imageserver we must make sure to apply it on every app this is where I think a pool make sense.
  • Another situation is when it scale down it evict the kube-system service that is on the node and it restart else where in case of heapster we lose 5 min of metrics. Not a big deal but we might end up with lots of gap in our graph
@scjodyscjodySep 24, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just suggesting using podAntiAffinity to prevent two imageserver pods from occupying the same node like we do now.
I still don't understand why having Kubernetes internal pods on the same nodes as the imageserver pods is a big deal. They don't use significant amounts of resources, do they? I do understand your concern about losing metrics, but I think it's worth trying anyway. I'm surprised Kubernetes isn't designed to scale down by terminating other nodes rather than these, but it sounds like something we have to live with.
@mag009mag009Sep 24, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed it back the way it was to use the default-pool and re-added the AntiAffinity to prevent two imageserver on the same host.
  • I've re-ran a stress test and no issue with heapster.
 affinity: 
 nodeAffinity: 
 requiredDuringSchedulingIgnoredDuringExecution: 
 nodeSelectorTerms: 
 - matchExpressions: 
 - key: failure-domain.beta.kubernetes.io/zone 
 operator: In 
 values: 
 - us-central1-a 
 - us-central1-b 
 - us-central1-c
 podAntiAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 preferredDuringSchedulingIgnoredDuringExecution:
 - podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: "app"
 operator: In
 values:
 - imageserver
 topologyKey: "kubernetes.io/hostname"
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: failure-domain.beta.kubernetes.io/zone
 weight: 1
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: kubernetes.io/hostname
 containers:
 - name: imageserver-app
 image: quay.io/plotly/image-exporter:master
Expand All@@ -40,6 +65,12 @@ spec:
 # starting the pod. This is useful when debugging, but should be turned
 # off in production.
 imagePullPolicy: Always
 resources: 
 limits: 
 memory: 2396Mi 
Comment thread
mag009 marked this conversation as resolved.

 requests: 
 cpu: 100m
 memory: 1Gi
 ports:
 - name: http-server
 containerPort: 9091
Expand Down

6 changes: 3 additions & 3 deletions
deployment/kube/prod/hpa.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -12,7 +12,7 @@ spec:
 kind: Deployment
 name: imageserver
 # Set this to 3x "min-nodes":
 minReplicas: 12
 minReplicas: 3
 # Set this to 3x "max-nodes":
 maxReplicas: 12
 targetCPUUtilizationPercentage: 50
 maxReplicas: 18
 targetCPUUtilizationPercentage: 30

9 changes: 9 additions & 0 deletions
deployment/kube/prod/pdb.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
 name: imageserver-pdb
spec:
 minAvailable: 2
 selector:
 matchLabels:
 app: imageserver

47 changes: 40 additions & 7 deletions
deployment/kube/stage/frontend.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -5,27 +5,54 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
 annotations:
 cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
 name: imageserver
 labels:
 app: imageserver
spec:
 replicas: 3
 strategy:
 rollingUpdate:
 maxSurge: 100%
 maxUnavailable: 25%
 type: RollingUpdate
 template:
 metadata:
 labels:
 app: imageserver
 tier: frontend
 spec:
 affinity:
podAntiAffinity:
nodeAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 nodeSelectorTerms:
 - matchExpressions:
 - key: failure-domain.beta.kubernetes.io/zone
 operator: In
 values:
 - us-central1-a
 - us-central1-b
 - us-central1-c
 podAntiAffinity:
 preferredDuringSchedulingIgnoredDuringExecution:
 - podAffinityTerm:
 labelSelector:
 matchExpressions:
 - key: "app"
 operator: In
 values:
 - imageserver
 topologyKey: "kubernetes.io/hostname"
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: failure-domain.beta.kubernetes.io/zone
 weight: 1
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchExpressions:
 - key: app
 operator: In
 values:
 - imageserver
 topologyKey: kubernetes.io/hostname
 containers:
 - name: imageserver-app
 image: quay.io/plotly/image-exporter:master
Expand All@@ -43,6 +70,12 @@ spec:
 ports:
 - name: http-server
 containerPort: 9091
 resources:
 limits:
 memory: 2396Mi
 requests:
 cpu: 100m
 memory: 1Gi
 volumeMounts:
 - mountPath: "/usr/share/fonts/user"
 name: plotly-cloud-licensed-fonts
Expand Down

4 changes: 2 additions & 2 deletions
deployment/kube/stage/hpa.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -14,5 +14,5 @@ spec:
 # Set this to 3x "min-nodes":
 minReplicas: 3
 # Set this to 3x "max-nodes":
 maxReplicas: 3
 targetCPUUtilizationPercentage: 50
 maxReplicas: 6
@scjodyscjodySep 13, 2018

Copy link


Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either the comment needs to be updated, or something else...
@mag009mag009Sep 13, 2018

Copy link


Copy Markdown

ContributorAuthor
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

noted!
 targetCPUUtilizationPercentage: 30

9 changes: 9 additions & 0 deletions
deployment/kube/stage/pdb.yaml

View file


Open in desktop

Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
 name: imageserver-pdb
spec:
 minAvailable: 2
Comment thread
mag009 marked this conversation as resolved.

 selector:
 matchLabels:
 app: imageserver

2 changes: 1 addition & 1 deletion
deployment/run_server

View file


Open in desktop

Original file line numberDiff line numberDiff line change
Expand Up@@ -26,7 +26,7 @@ fi
pkill Xvfb
pkill node

xvfb-run --auto-servernum --server-args '-screen 0 640x480x24' ./bin/orca.js serve --request-limit=1000 --safe-mode $PLOTLYJS_ARG $@ 1>/proc/1/fd/1 2>/proc/1/fd/2 &
xvfb-run --auto-servernum --server-args '-screen 0 640x480x24' ./bin/orca.js serve --safe-mode $PLOTLYJS_ARG $@ 1>/proc/1/fd/1 2>/proc/1/fd/2 &
echo \$! > \$PIDFILE

EOF
Expand Down