Build --secret with buildkit - #1288

Merged
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets
Aug 17, 2018
Merged

Build --secret with buildkit#1288
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets

Conversation

@tiborvass

@tiborvasstiborvass commented Aug 14, 2018

Copy link
Copy Markdown
Collaborator

This patch implements docker build --secret id=mysecret,src=/secret/file
for buildkit frontends that request the mysecret secret.

It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret

Signed-off-by: Tibor Vass tibor@docker.com

Also revendors buildkit and docker/docker

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester@cpuguy83 :) this is pretty dope. This is just wiring stuff together, the heavy lifting was done in buildkit repo.

@vdemeestervdemeester left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🐯
Small lint failure to fix though 😉

cli/command/image/build_buildkit.go:1::warning: file is not gofmted with -s (gofmt)
cli/command/image/build_buildkit.go:1::warning: file is not goimported (goimports)

@codecov-io

codecov-io commented Aug 14, 2018

Copy link
Copy Markdown

Codecov Report

Merging #1288 into master will increase coverage by 0.01%.
The diff coverage is 60.46%.

@@ Coverage Diff @@## master #1288 +/- ##
==========================================
+ Coverage 54.03% 54.05% +0.01% 
==========================================
Files 272 272 Lines 18072 18114 +42 ==========================================
+ Hits 9766 9792 +26 - Misses 7690 7706 +16 
Partials 616 616

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester fixed

@cpuguy83cpuguy83 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

return secretsprovider.NewSecretProvider(store), nil
}

func parseSecret(value string) (*secretsprovider.FileSource, error) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can add some test for these two new functions?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@thaJeztah

Copy link
Copy Markdown
Member

Trying to get this to work, but I'm probably doing it wrong (running against Docker 18.06)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --console=false --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:runmount20180618FROM busyboxRUN echo "hello world"RUN --mount=type=secret,id=mysecret echo "anything here"RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF

Whatever I try to do with --mount=type=secret.... seems to give me an exit code 2

@cpuguy83

Copy link
Copy Markdown
Collaborator

It doesn't work on 18.06, missing some daemon stuff.

@AkihiroSuda

Copy link
Copy Markdown
Collaborator

Is # syntax = still needed?

@tiborvass

tiborvass commented Aug 17, 2018

Copy link
Copy Markdown
CollaboratorAuthor

@thaJeztah you're using the runmount flavored dockefile frontend, instead of secrets.

Note that --secret is now guarded by API version 1.39.

@cpuguy83

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

Agreed it's weird, will debug it but shouldn't be a blocker for this PR. I added a couple of tests.

@AkihiroSuda

Is # syntax = still needed?

Yes, this is not part of the stable compiled-in default frontend.

PTAL :)

This patch implements `docker build --secret id=mysecret,src=/secret/file`
for buildkit frontends that request the mysecret secret.
It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret
Signed-off-by: Tibor Vass <tibor@docker.com>
vendors github.com/docker/docker to a7ff19d69a90dfe152abd146221c8b9b46a0903d
Signed-off-by: Tibor Vass <tibor@docker.com>
@thaJeztah

Copy link
Copy Markdown
Member

Tried this again, and looks good :)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --progress=plain --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:secrets20180808FROM busyboxRUN --mount=type=secret,id=mysecret cat /run/secrets/mysecretRUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF
#8 [2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
#8 digest: sha256:ab7659590ba7fcf4ec338708fcf824ea76ee7b48b13dce4b9c01faf19395c39d
#8 name: "[2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret"
#8 started: 2018-08-17 16:08:28.0910424 +0000 UTC
#8 0.303 hello secret#8 completed: 2018-08-17 16:08:28.6508443 +0000 UTC
#8 duration: 559.8019ms
#8 0.303 hello secret
#9 [3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar
#9 digest: sha256:67234c5c8ff0bbd7d6c342980f66f9e5ac75405d2860bbd15747e44291b5cb1d
#9 name: "[3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar"
#9 started: 2018-08-17 16:08:28.6597585 +0000 UTC
#9 0.343 hello secret#9 completed: 2018-08-17 16:08:29.2602578 +0000 UTC
#9 duration: 600.4993ms
#9 0.343 hello secret

@thaJeztahthaJeztah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah

Copy link
Copy Markdown
Member

/cc @albers for bash completion 😅

@thaJeztah
thaJeztah merged commit cb142fa into docker:masterAug 17, 2018
@GordonTheTurtleGordonTheTurtle added this to the 18.09.0 milestone Aug 17, 2018
@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

Big thanks to y'all for the review and to Tonis for the implementation!

chrishiestand added a commit to chrishiestand/docker-image-resource that referenced this pull request Aug 22, 2018
It is inaccurate to say that build args will not persist in the final image. They are visible with `docker history` and `docker inspect`.
As soon as possible, we should upgrade to docker 18.09 to support `--secret` - a way to securely pass credentials into the build context.
docker/cli#1288
@haizaar

Copy link
Copy Markdown

@thaJeztah
Can you please elaborate on export DOCKER_BUILDKIT=1?
Will we have to define this env var when this feature becomes available as part of docker-ce 18.09?

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@tiborvass@codecov-io@thaJeztah@cpuguy83@AkihiroSuda@haizaar@vdemeester@GordonTheTurtle
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Build --secret with buildkit - #1288

Merged
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets
Aug 17, 2018
Merged

Build --secret with buildkit#1288
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets

Conversation

@tiborvass

@tiborvasstiborvass commented Aug 14, 2018

Copy link
Copy Markdown
Collaborator

This patch implements docker build --secret id=mysecret,src=/secret/file
for buildkit frontends that request the mysecret secret.

It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret

Signed-off-by: Tibor Vass tibor@docker.com

Also revendors buildkit and docker/docker

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester@cpuguy83 :) this is pretty dope. This is just wiring stuff together, the heavy lifting was done in buildkit repo.

@vdemeestervdemeester left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🐯
Small lint failure to fix though 😉

cli/command/image/build_buildkit.go:1::warning: file is not gofmted with -s (gofmt)
cli/command/image/build_buildkit.go:1::warning: file is not goimported (goimports)

@codecov-io

codecov-io commented Aug 14, 2018

Copy link
Copy Markdown

Codecov Report

Merging #1288 into master will increase coverage by 0.01%.
The diff coverage is 60.46%.

@@ Coverage Diff @@## master #1288 +/- ##
==========================================
+ Coverage 54.03% 54.05% +0.01% 
==========================================
Files 272 272 Lines 18072 18114 +42 ==========================================
+ Hits 9766 9792 +26 - Misses 7690 7706 +16 
Partials 616 616

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester fixed

@cpuguy83cpuguy83 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

return secretsprovider.NewSecretProvider(store), nil
}

func parseSecret(value string) (*secretsprovider.FileSource, error) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can add some test for these two new functions?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@thaJeztah

Copy link
Copy Markdown
Member

Trying to get this to work, but I'm probably doing it wrong (running against Docker 18.06)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --console=false --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:runmount20180618FROM busyboxRUN echo "hello world"RUN --mount=type=secret,id=mysecret echo "anything here"RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF

Whatever I try to do with --mount=type=secret.... seems to give me an exit code 2

@cpuguy83

Copy link
Copy Markdown
Collaborator

It doesn't work on 18.06, missing some daemon stuff.

@AkihiroSuda

Copy link
Copy Markdown
Collaborator

Is # syntax = still needed?

@tiborvass

tiborvass commented Aug 17, 2018

Copy link
Copy Markdown
CollaboratorAuthor

@thaJeztah you're using the runmount flavored dockefile frontend, instead of secrets.

Note that --secret is now guarded by API version 1.39.

@cpuguy83

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

Agreed it's weird, will debug it but shouldn't be a blocker for this PR. I added a couple of tests.

@AkihiroSuda

Is # syntax = still needed?

Yes, this is not part of the stable compiled-in default frontend.

PTAL :)

This patch implements `docker build --secret id=mysecret,src=/secret/file`
for buildkit frontends that request the mysecret secret.
It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret
Signed-off-by: Tibor Vass <tibor@docker.com>
vendors github.com/docker/docker to a7ff19d69a90dfe152abd146221c8b9b46a0903d
Signed-off-by: Tibor Vass <tibor@docker.com>
@thaJeztah

Copy link
Copy Markdown
Member

Tried this again, and looks good :)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --progress=plain --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:secrets20180808FROM busyboxRUN --mount=type=secret,id=mysecret cat /run/secrets/mysecretRUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF
#8 [2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
#8 digest: sha256:ab7659590ba7fcf4ec338708fcf824ea76ee7b48b13dce4b9c01faf19395c39d
#8 name: "[2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret"
#8 started: 2018-08-17 16:08:28.0910424 +0000 UTC
#8 0.303 hello secret#8 completed: 2018-08-17 16:08:28.6508443 +0000 UTC
#8 duration: 559.8019ms
#8 0.303 hello secret
#9 [3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar
#9 digest: sha256:67234c5c8ff0bbd7d6c342980f66f9e5ac75405d2860bbd15747e44291b5cb1d
#9 name: "[3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar"
#9 started: 2018-08-17 16:08:28.6597585 +0000 UTC
#9 0.343 hello secret#9 completed: 2018-08-17 16:08:29.2602578 +0000 UTC
#9 duration: 600.4993ms
#9 0.343 hello secret

@thaJeztahthaJeztah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah

Copy link
Copy Markdown
Member

/cc @albers for bash completion 😅

@thaJeztah
thaJeztah merged commit cb142fa into docker:masterAug 17, 2018
@GordonTheTurtleGordonTheTurtle added this to the 18.09.0 milestone Aug 17, 2018
@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

Big thanks to y'all for the review and to Tonis for the implementation!

chrishiestand added a commit to chrishiestand/docker-image-resource that referenced this pull request Aug 22, 2018
It is inaccurate to say that build args will not persist in the final image. They are visible with `docker history` and `docker inspect`.
As soon as possible, we should upgrade to docker 18.09 to support `--secret` - a way to securely pass credentials into the build context.
docker/cli#1288
@haizaar

Copy link
Copy Markdown

@thaJeztah
Can you please elaborate on export DOCKER_BUILDKIT=1?
Will we have to define this env var when this feature becomes available as part of docker-ce 18.09?

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@tiborvass@codecov-io@thaJeztah@cpuguy83@AkihiroSuda@haizaar@vdemeester@GordonTheTurtle
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Build --secret with buildkit - #1288

Merged
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets
Aug 17, 2018
Merged

Build --secret with buildkit#1288
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets

Conversation

@tiborvass

@tiborvasstiborvass commented Aug 14, 2018

Copy link
Copy Markdown
Collaborator

This patch implements docker build --secret id=mysecret,src=/secret/file
for buildkit frontends that request the mysecret secret.

It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret

Signed-off-by: Tibor Vass tibor@docker.com

Also revendors buildkit and docker/docker

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester@cpuguy83 :) this is pretty dope. This is just wiring stuff together, the heavy lifting was done in buildkit repo.

@vdemeestervdemeester left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🐯
Small lint failure to fix though 😉

cli/command/image/build_buildkit.go:1::warning: file is not gofmted with -s (gofmt)
cli/command/image/build_buildkit.go:1::warning: file is not goimported (goimports)

@codecov-io

codecov-io commented Aug 14, 2018

Copy link
Copy Markdown

Codecov Report

Merging #1288 into master will increase coverage by 0.01%.
The diff coverage is 60.46%.

@@ Coverage Diff @@## master #1288 +/- ##
==========================================
+ Coverage 54.03% 54.05% +0.01% 
==========================================
Files 272 272 Lines 18072 18114 +42 ==========================================
+ Hits 9766 9792 +26 - Misses 7690 7706 +16 
Partials 616 616

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester fixed

@cpuguy83cpuguy83 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

return secretsprovider.NewSecretProvider(store), nil
}

func parseSecret(value string) (*secretsprovider.FileSource, error) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can add some test for these two new functions?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@thaJeztah

Copy link
Copy Markdown
Member

Trying to get this to work, but I'm probably doing it wrong (running against Docker 18.06)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --console=false --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:runmount20180618FROM busyboxRUN echo "hello world"RUN --mount=type=secret,id=mysecret echo "anything here"RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF

Whatever I try to do with --mount=type=secret.... seems to give me an exit code 2

@cpuguy83

Copy link
Copy Markdown
Collaborator

It doesn't work on 18.06, missing some daemon stuff.

@AkihiroSuda

Copy link
Copy Markdown
Collaborator

Is # syntax = still needed?

@tiborvass

tiborvass commented Aug 17, 2018

Copy link
Copy Markdown
CollaboratorAuthor

@thaJeztah you're using the runmount flavored dockefile frontend, instead of secrets.

Note that --secret is now guarded by API version 1.39.

@cpuguy83

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

Agreed it's weird, will debug it but shouldn't be a blocker for this PR. I added a couple of tests.

@AkihiroSuda

Is # syntax = still needed?

Yes, this is not part of the stable compiled-in default frontend.

PTAL :)

This patch implements `docker build --secret id=mysecret,src=/secret/file`
for buildkit frontends that request the mysecret secret.
It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret
Signed-off-by: Tibor Vass <tibor@docker.com>
vendors github.com/docker/docker to a7ff19d69a90dfe152abd146221c8b9b46a0903d
Signed-off-by: Tibor Vass <tibor@docker.com>
@thaJeztah

Copy link
Copy Markdown
Member

Tried this again, and looks good :)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --progress=plain --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:secrets20180808FROM busyboxRUN --mount=type=secret,id=mysecret cat /run/secrets/mysecretRUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF
#8 [2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
#8 digest: sha256:ab7659590ba7fcf4ec338708fcf824ea76ee7b48b13dce4b9c01faf19395c39d
#8 name: "[2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret"
#8 started: 2018-08-17 16:08:28.0910424 +0000 UTC
#8 0.303 hello secret#8 completed: 2018-08-17 16:08:28.6508443 +0000 UTC
#8 duration: 559.8019ms
#8 0.303 hello secret
#9 [3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar
#9 digest: sha256:67234c5c8ff0bbd7d6c342980f66f9e5ac75405d2860bbd15747e44291b5cb1d
#9 name: "[3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar"
#9 started: 2018-08-17 16:08:28.6597585 +0000 UTC
#9 0.343 hello secret#9 completed: 2018-08-17 16:08:29.2602578 +0000 UTC
#9 duration: 600.4993ms
#9 0.343 hello secret

@thaJeztahthaJeztah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah

Copy link
Copy Markdown
Member

/cc @albers for bash completion 😅

@thaJeztah
thaJeztah merged commit cb142fa into docker:masterAug 17, 2018
@GordonTheTurtleGordonTheTurtle added this to the 18.09.0 milestone Aug 17, 2018
@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

Big thanks to y'all for the review and to Tonis for the implementation!

chrishiestand added a commit to chrishiestand/docker-image-resource that referenced this pull request Aug 22, 2018
It is inaccurate to say that build args will not persist in the final image. They are visible with `docker history` and `docker inspect`.
As soon as possible, we should upgrade to docker 18.09 to support `--secret` - a way to securely pass credentials into the build context.
docker/cli#1288
@haizaar

Copy link
Copy Markdown

@thaJeztah
Can you please elaborate on export DOCKER_BUILDKIT=1?
Will we have to define this env var when this feature becomes available as part of docker-ce 18.09?

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@tiborvass@codecov-io@thaJeztah@cpuguy83@AkihiroSuda@haizaar@vdemeester@GordonTheTurtle
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Build --secret with buildkit - #1288

Merged
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets
Aug 17, 2018
Merged

Build --secret with buildkit#1288
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets

Conversation

@tiborvass

@tiborvasstiborvass commented Aug 14, 2018

Copy link
Copy Markdown
Collaborator

This patch implements docker build --secret id=mysecret,src=/secret/file
for buildkit frontends that request the mysecret secret.

It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret

Signed-off-by: Tibor Vass tibor@docker.com

Also revendors buildkit and docker/docker

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester@cpuguy83 :) this is pretty dope. This is just wiring stuff together, the heavy lifting was done in buildkit repo.

@vdemeestervdemeester left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🐯
Small lint failure to fix though 😉

cli/command/image/build_buildkit.go:1::warning: file is not gofmted with -s (gofmt)
cli/command/image/build_buildkit.go:1::warning: file is not goimported (goimports)

@codecov-io

codecov-io commented Aug 14, 2018

Copy link
Copy Markdown

Codecov Report

Merging #1288 into master will increase coverage by 0.01%.
The diff coverage is 60.46%.

@@ Coverage Diff @@## master #1288 +/- ##
==========================================
+ Coverage 54.03% 54.05% +0.01% 
==========================================
Files 272 272 Lines 18072 18114 +42 ==========================================
+ Hits 9766 9792 +26 - Misses 7690 7706 +16 
Partials 616 616

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester fixed

@cpuguy83cpuguy83 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

return secretsprovider.NewSecretProvider(store), nil
}

func parseSecret(value string) (*secretsprovider.FileSource, error) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can add some test for these two new functions?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@thaJeztah

Copy link
Copy Markdown
Member

Trying to get this to work, but I'm probably doing it wrong (running against Docker 18.06)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --console=false --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:runmount20180618FROM busyboxRUN echo "hello world"RUN --mount=type=secret,id=mysecret echo "anything here"RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF

Whatever I try to do with --mount=type=secret.... seems to give me an exit code 2

@cpuguy83

Copy link
Copy Markdown
Collaborator

It doesn't work on 18.06, missing some daemon stuff.

@AkihiroSuda

Copy link
Copy Markdown
Collaborator

Is # syntax = still needed?

@tiborvass

tiborvass commented Aug 17, 2018

Copy link
Copy Markdown
CollaboratorAuthor

@thaJeztah you're using the runmount flavored dockefile frontend, instead of secrets.

Note that --secret is now guarded by API version 1.39.

@cpuguy83

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

Agreed it's weird, will debug it but shouldn't be a blocker for this PR. I added a couple of tests.

@AkihiroSuda

Is # syntax = still needed?

Yes, this is not part of the stable compiled-in default frontend.

PTAL :)

This patch implements `docker build --secret id=mysecret,src=/secret/file`
for buildkit frontends that request the mysecret secret.
It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret
Signed-off-by: Tibor Vass <tibor@docker.com>
vendors github.com/docker/docker to a7ff19d69a90dfe152abd146221c8b9b46a0903d
Signed-off-by: Tibor Vass <tibor@docker.com>
@thaJeztah

Copy link
Copy Markdown
Member

Tried this again, and looks good :)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --progress=plain --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:secrets20180808FROM busyboxRUN --mount=type=secret,id=mysecret cat /run/secrets/mysecretRUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF
#8 [2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
#8 digest: sha256:ab7659590ba7fcf4ec338708fcf824ea76ee7b48b13dce4b9c01faf19395c39d
#8 name: "[2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret"
#8 started: 2018-08-17 16:08:28.0910424 +0000 UTC
#8 0.303 hello secret#8 completed: 2018-08-17 16:08:28.6508443 +0000 UTC
#8 duration: 559.8019ms
#8 0.303 hello secret
#9 [3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar
#9 digest: sha256:67234c5c8ff0bbd7d6c342980f66f9e5ac75405d2860bbd15747e44291b5cb1d
#9 name: "[3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar"
#9 started: 2018-08-17 16:08:28.6597585 +0000 UTC
#9 0.343 hello secret#9 completed: 2018-08-17 16:08:29.2602578 +0000 UTC
#9 duration: 600.4993ms
#9 0.343 hello secret

@thaJeztahthaJeztah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah

Copy link
Copy Markdown
Member

/cc @albers for bash completion 😅

@thaJeztah
thaJeztah merged commit cb142fa into docker:masterAug 17, 2018
@GordonTheTurtleGordonTheTurtle added this to the 18.09.0 milestone Aug 17, 2018
@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

Big thanks to y'all for the review and to Tonis for the implementation!

chrishiestand added a commit to chrishiestand/docker-image-resource that referenced this pull request Aug 22, 2018
It is inaccurate to say that build args will not persist in the final image. They are visible with `docker history` and `docker inspect`.
As soon as possible, we should upgrade to docker 18.09 to support `--secret` - a way to securely pass credentials into the build context.
docker/cli#1288
@haizaar

Copy link
Copy Markdown

@thaJeztah
Can you please elaborate on export DOCKER_BUILDKIT=1?
Will we have to define this env var when this feature becomes available as part of docker-ce 18.09?

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@tiborvass@codecov-io@thaJeztah@cpuguy83@AkihiroSuda@haizaar@vdemeester@GordonTheTurtle
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Build --secret with buildkit - #1288

Merged
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets
Aug 17, 2018
Merged

Build --secret with buildkit#1288
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets

Conversation

@tiborvass

@tiborvasstiborvass commented Aug 14, 2018

Copy link
Copy Markdown
Collaborator

This patch implements docker build --secret id=mysecret,src=/secret/file
for buildkit frontends that request the mysecret secret.

It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret

Signed-off-by: Tibor Vass tibor@docker.com

Also revendors buildkit and docker/docker

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester@cpuguy83 :) this is pretty dope. This is just wiring stuff together, the heavy lifting was done in buildkit repo.

@vdemeestervdemeester left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🐯
Small lint failure to fix though 😉

cli/command/image/build_buildkit.go:1::warning: file is not gofmted with -s (gofmt)
cli/command/image/build_buildkit.go:1::warning: file is not goimported (goimports)

@codecov-io

codecov-io commented Aug 14, 2018

Copy link
Copy Markdown

Codecov Report

Merging #1288 into master will increase coverage by 0.01%.
The diff coverage is 60.46%.

@@ Coverage Diff @@## master #1288 +/- ##
==========================================
+ Coverage 54.03% 54.05% +0.01% 
==========================================
Files 272 272 Lines 18072 18114 +42 ==========================================
+ Hits 9766 9792 +26 - Misses 7690 7706 +16 
Partials 616 616

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester fixed

@cpuguy83cpuguy83 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

return secretsprovider.NewSecretProvider(store), nil
}

func parseSecret(value string) (*secretsprovider.FileSource, error) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can add some test for these two new functions?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@thaJeztah

Copy link
Copy Markdown
Member

Trying to get this to work, but I'm probably doing it wrong (running against Docker 18.06)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --console=false --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:runmount20180618FROM busyboxRUN echo "hello world"RUN --mount=type=secret,id=mysecret echo "anything here"RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF

Whatever I try to do with --mount=type=secret.... seems to give me an exit code 2

@cpuguy83

Copy link
Copy Markdown
Collaborator

It doesn't work on 18.06, missing some daemon stuff.

@AkihiroSuda

Copy link
Copy Markdown
Collaborator

Is # syntax = still needed?

@tiborvass

tiborvass commented Aug 17, 2018

Copy link
Copy Markdown
CollaboratorAuthor

@thaJeztah you're using the runmount flavored dockefile frontend, instead of secrets.

Note that --secret is now guarded by API version 1.39.

@cpuguy83

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

Agreed it's weird, will debug it but shouldn't be a blocker for this PR. I added a couple of tests.

@AkihiroSuda

Is # syntax = still needed?

Yes, this is not part of the stable compiled-in default frontend.

PTAL :)

This patch implements `docker build --secret id=mysecret,src=/secret/file`
for buildkit frontends that request the mysecret secret.
It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret
Signed-off-by: Tibor Vass <tibor@docker.com>
vendors github.com/docker/docker to a7ff19d69a90dfe152abd146221c8b9b46a0903d
Signed-off-by: Tibor Vass <tibor@docker.com>
@thaJeztah

Copy link
Copy Markdown
Member

Tried this again, and looks good :)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --progress=plain --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:secrets20180808FROM busyboxRUN --mount=type=secret,id=mysecret cat /run/secrets/mysecretRUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF
#8 [2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
#8 digest: sha256:ab7659590ba7fcf4ec338708fcf824ea76ee7b48b13dce4b9c01faf19395c39d
#8 name: "[2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret"
#8 started: 2018-08-17 16:08:28.0910424 +0000 UTC
#8 0.303 hello secret#8 completed: 2018-08-17 16:08:28.6508443 +0000 UTC
#8 duration: 559.8019ms
#8 0.303 hello secret
#9 [3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar
#9 digest: sha256:67234c5c8ff0bbd7d6c342980f66f9e5ac75405d2860bbd15747e44291b5cb1d
#9 name: "[3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar"
#9 started: 2018-08-17 16:08:28.6597585 +0000 UTC
#9 0.343 hello secret#9 completed: 2018-08-17 16:08:29.2602578 +0000 UTC
#9 duration: 600.4993ms
#9 0.343 hello secret

@thaJeztahthaJeztah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah

Copy link
Copy Markdown
Member

/cc @albers for bash completion 😅

@thaJeztah
thaJeztah merged commit cb142fa into docker:masterAug 17, 2018
@GordonTheTurtleGordonTheTurtle added this to the 18.09.0 milestone Aug 17, 2018
@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

Big thanks to y'all for the review and to Tonis for the implementation!

chrishiestand added a commit to chrishiestand/docker-image-resource that referenced this pull request Aug 22, 2018
It is inaccurate to say that build args will not persist in the final image. They are visible with `docker history` and `docker inspect`.
As soon as possible, we should upgrade to docker 18.09 to support `--secret` - a way to securely pass credentials into the build context.
docker/cli#1288
@haizaar

Copy link
Copy Markdown

@thaJeztah
Can you please elaborate on export DOCKER_BUILDKIT=1?
Will we have to define this env var when this feature becomes available as part of docker-ce 18.09?

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@tiborvass@codecov-io@thaJeztah@cpuguy83@AkihiroSuda@haizaar@vdemeester@GordonTheTurtle
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Build --secret with buildkit - #1288

Merged
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets
Aug 17, 2018
Merged

Build --secret with buildkit#1288
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets

Conversation

@tiborvass

@tiborvasstiborvass commented Aug 14, 2018

Copy link
Copy Markdown
Collaborator

This patch implements docker build --secret id=mysecret,src=/secret/file
for buildkit frontends that request the mysecret secret.

It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret

Signed-off-by: Tibor Vass tibor@docker.com

Also revendors buildkit and docker/docker

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester@cpuguy83 :) this is pretty dope. This is just wiring stuff together, the heavy lifting was done in buildkit repo.

@vdemeestervdemeester left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🐯
Small lint failure to fix though 😉

cli/command/image/build_buildkit.go:1::warning: file is not gofmted with -s (gofmt)
cli/command/image/build_buildkit.go:1::warning: file is not goimported (goimports)

@codecov-io

codecov-io commented Aug 14, 2018

Copy link
Copy Markdown

Codecov Report

Merging #1288 into master will increase coverage by 0.01%.
The diff coverage is 60.46%.

@@ Coverage Diff @@## master #1288 +/- ##
==========================================
+ Coverage 54.03% 54.05% +0.01% 
==========================================
Files 272 272 Lines 18072 18114 +42 ==========================================
+ Hits 9766 9792 +26 - Misses 7690 7706 +16 
Partials 616 616

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester fixed

@cpuguy83cpuguy83 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

return secretsprovider.NewSecretProvider(store), nil
}

func parseSecret(value string) (*secretsprovider.FileSource, error) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can add some test for these two new functions?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@thaJeztah

Copy link
Copy Markdown
Member

Trying to get this to work, but I'm probably doing it wrong (running against Docker 18.06)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --console=false --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:runmount20180618FROM busyboxRUN echo "hello world"RUN --mount=type=secret,id=mysecret echo "anything here"RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF

Whatever I try to do with --mount=type=secret.... seems to give me an exit code 2

@cpuguy83

Copy link
Copy Markdown
Collaborator

It doesn't work on 18.06, missing some daemon stuff.

@AkihiroSuda

Copy link
Copy Markdown
Collaborator

Is # syntax = still needed?

@tiborvass

tiborvass commented Aug 17, 2018

Copy link
Copy Markdown
CollaboratorAuthor

@thaJeztah you're using the runmount flavored dockefile frontend, instead of secrets.

Note that --secret is now guarded by API version 1.39.

@cpuguy83

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

Agreed it's weird, will debug it but shouldn't be a blocker for this PR. I added a couple of tests.

@AkihiroSuda

Is # syntax = still needed?

Yes, this is not part of the stable compiled-in default frontend.

PTAL :)

This patch implements `docker build --secret id=mysecret,src=/secret/file`
for buildkit frontends that request the mysecret secret.
It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret
Signed-off-by: Tibor Vass <tibor@docker.com>
vendors github.com/docker/docker to a7ff19d69a90dfe152abd146221c8b9b46a0903d
Signed-off-by: Tibor Vass <tibor@docker.com>
@thaJeztah

Copy link
Copy Markdown
Member

Tried this again, and looks good :)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --progress=plain --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:secrets20180808FROM busyboxRUN --mount=type=secret,id=mysecret cat /run/secrets/mysecretRUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF
#8 [2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
#8 digest: sha256:ab7659590ba7fcf4ec338708fcf824ea76ee7b48b13dce4b9c01faf19395c39d
#8 name: "[2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret"
#8 started: 2018-08-17 16:08:28.0910424 +0000 UTC
#8 0.303 hello secret#8 completed: 2018-08-17 16:08:28.6508443 +0000 UTC
#8 duration: 559.8019ms
#8 0.303 hello secret
#9 [3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar
#9 digest: sha256:67234c5c8ff0bbd7d6c342980f66f9e5ac75405d2860bbd15747e44291b5cb1d
#9 name: "[3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar"
#9 started: 2018-08-17 16:08:28.6597585 +0000 UTC
#9 0.343 hello secret#9 completed: 2018-08-17 16:08:29.2602578 +0000 UTC
#9 duration: 600.4993ms
#9 0.343 hello secret

@thaJeztahthaJeztah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah

Copy link
Copy Markdown
Member

/cc @albers for bash completion 😅

@thaJeztah
thaJeztah merged commit cb142fa into docker:masterAug 17, 2018
@GordonTheTurtleGordonTheTurtle added this to the 18.09.0 milestone Aug 17, 2018
@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

Big thanks to y'all for the review and to Tonis for the implementation!

chrishiestand added a commit to chrishiestand/docker-image-resource that referenced this pull request Aug 22, 2018
It is inaccurate to say that build args will not persist in the final image. They are visible with `docker history` and `docker inspect`.
As soon as possible, we should upgrade to docker 18.09 to support `--secret` - a way to securely pass credentials into the build context.
docker/cli#1288
@haizaar

Copy link
Copy Markdown

@thaJeztah
Can you please elaborate on export DOCKER_BUILDKIT=1?
Will we have to define this env var when this feature becomes available as part of docker-ce 18.09?

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@tiborvass@codecov-io@thaJeztah@cpuguy83@AkihiroSuda@haizaar@vdemeester@GordonTheTurtle
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Build --secret with buildkit - #1288

Merged
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets
Aug 17, 2018
Merged

Build --secret with buildkit#1288
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets

Conversation

@tiborvass

@tiborvasstiborvass commented Aug 14, 2018

Copy link
Copy Markdown
Collaborator

This patch implements docker build --secret id=mysecret,src=/secret/file
for buildkit frontends that request the mysecret secret.

It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret

Signed-off-by: Tibor Vass tibor@docker.com

Also revendors buildkit and docker/docker

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester@cpuguy83 :) this is pretty dope. This is just wiring stuff together, the heavy lifting was done in buildkit repo.

@vdemeestervdemeester left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🐯
Small lint failure to fix though 😉

cli/command/image/build_buildkit.go:1::warning: file is not gofmted with -s (gofmt)
cli/command/image/build_buildkit.go:1::warning: file is not goimported (goimports)

@codecov-io

codecov-io commented Aug 14, 2018

Copy link
Copy Markdown

Codecov Report

Merging #1288 into master will increase coverage by 0.01%.
The diff coverage is 60.46%.

@@ Coverage Diff @@## master #1288 +/- ##
==========================================
+ Coverage 54.03% 54.05% +0.01% 
==========================================
Files 272 272 Lines 18072 18114 +42 ==========================================
+ Hits 9766 9792 +26 - Misses 7690 7706 +16 
Partials 616 616

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester fixed

@cpuguy83cpuguy83 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

return secretsprovider.NewSecretProvider(store), nil
}

func parseSecret(value string) (*secretsprovider.FileSource, error) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can add some test for these two new functions?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@thaJeztah

Copy link
Copy Markdown
Member

Trying to get this to work, but I'm probably doing it wrong (running against Docker 18.06)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --console=false --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:runmount20180618FROM busyboxRUN echo "hello world"RUN --mount=type=secret,id=mysecret echo "anything here"RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF

Whatever I try to do with --mount=type=secret.... seems to give me an exit code 2

@cpuguy83

Copy link
Copy Markdown
Collaborator

It doesn't work on 18.06, missing some daemon stuff.

@AkihiroSuda

Copy link
Copy Markdown
Collaborator

Is # syntax = still needed?

@tiborvass

tiborvass commented Aug 17, 2018

Copy link
Copy Markdown
CollaboratorAuthor

@thaJeztah you're using the runmount flavored dockefile frontend, instead of secrets.

Note that --secret is now guarded by API version 1.39.

@cpuguy83

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

Agreed it's weird, will debug it but shouldn't be a blocker for this PR. I added a couple of tests.

@AkihiroSuda

Is # syntax = still needed?

Yes, this is not part of the stable compiled-in default frontend.

PTAL :)

This patch implements `docker build --secret id=mysecret,src=/secret/file`
for buildkit frontends that request the mysecret secret.
It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret
Signed-off-by: Tibor Vass <tibor@docker.com>
vendors github.com/docker/docker to a7ff19d69a90dfe152abd146221c8b9b46a0903d
Signed-off-by: Tibor Vass <tibor@docker.com>
@thaJeztah

Copy link
Copy Markdown
Member

Tried this again, and looks good :)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --progress=plain --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:secrets20180808FROM busyboxRUN --mount=type=secret,id=mysecret cat /run/secrets/mysecretRUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF
#8 [2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
#8 digest: sha256:ab7659590ba7fcf4ec338708fcf824ea76ee7b48b13dce4b9c01faf19395c39d
#8 name: "[2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret"
#8 started: 2018-08-17 16:08:28.0910424 +0000 UTC
#8 0.303 hello secret#8 completed: 2018-08-17 16:08:28.6508443 +0000 UTC
#8 duration: 559.8019ms
#8 0.303 hello secret
#9 [3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar
#9 digest: sha256:67234c5c8ff0bbd7d6c342980f66f9e5ac75405d2860bbd15747e44291b5cb1d
#9 name: "[3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar"
#9 started: 2018-08-17 16:08:28.6597585 +0000 UTC
#9 0.343 hello secret#9 completed: 2018-08-17 16:08:29.2602578 +0000 UTC
#9 duration: 600.4993ms
#9 0.343 hello secret

@thaJeztahthaJeztah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah

Copy link
Copy Markdown
Member

/cc @albers for bash completion 😅

@thaJeztah
thaJeztah merged commit cb142fa into docker:masterAug 17, 2018
@GordonTheTurtleGordonTheTurtle added this to the 18.09.0 milestone Aug 17, 2018
@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

Big thanks to y'all for the review and to Tonis for the implementation!

chrishiestand added a commit to chrishiestand/docker-image-resource that referenced this pull request Aug 22, 2018
It is inaccurate to say that build args will not persist in the final image. They are visible with `docker history` and `docker inspect`.
As soon as possible, we should upgrade to docker 18.09 to support `--secret` - a way to securely pass credentials into the build context.
docker/cli#1288
@haizaar

Copy link
Copy Markdown

@thaJeztah
Can you please elaborate on export DOCKER_BUILDKIT=1?
Will we have to define this env var when this feature becomes available as part of docker-ce 18.09?

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@tiborvass@codecov-io@thaJeztah@cpuguy83@AkihiroSuda@haizaar@vdemeester@GordonTheTurtle
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Build --secret with buildkit - #1288

Merged
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets
Aug 17, 2018
Merged

Build --secret with buildkit#1288
thaJeztah merged 2 commits into
docker:masterfrom
tiborvass:build-secrets

Conversation

@tiborvass

@tiborvasstiborvass commented Aug 14, 2018

Copy link
Copy Markdown
Collaborator

This patch implements docker build --secret id=mysecret,src=/secret/file
for buildkit frontends that request the mysecret secret.

It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret

Signed-off-by: Tibor Vass tibor@docker.com

Also revendors buildkit and docker/docker

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester@cpuguy83 :) this is pretty dope. This is just wiring stuff together, the heavy lifting was done in buildkit repo.

@vdemeestervdemeester left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🐯
Small lint failure to fix though 😉

cli/command/image/build_buildkit.go:1::warning: file is not gofmted with -s (gofmt)
cli/command/image/build_buildkit.go:1::warning: file is not goimported (goimports)

@codecov-io

codecov-io commented Aug 14, 2018

Copy link
Copy Markdown

Codecov Report

Merging #1288 into master will increase coverage by 0.01%.
The diff coverage is 60.46%.

@@ Coverage Diff @@## master #1288 +/- ##
==========================================
+ Coverage 54.03% 54.05% +0.01% 
==========================================
Files 272 272 Lines 18072 18114 +42 ==========================================
+ Hits 9766 9792 +26 - Misses 7690 7706 +16 
Partials 616 616

@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

@vdemeester fixed

@cpuguy83cpuguy83 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

return secretsprovider.NewSecretProvider(store), nil
}

func parseSecret(value string) (*secretsprovider.FileSource, error) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can add some test for these two new functions?

Copy link
Copy Markdown
CollaboratorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@thaJeztah

Copy link
Copy Markdown
Member

Trying to get this to work, but I'm probably doing it wrong (running against Docker 18.06)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --console=false --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:runmount20180618FROM busyboxRUN echo "hello world"RUN --mount=type=secret,id=mysecret echo "anything here"RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF

Whatever I try to do with --mount=type=secret.... seems to give me an exit code 2

@cpuguy83

Copy link
Copy Markdown
Collaborator

It doesn't work on 18.06, missing some daemon stuff.

@AkihiroSuda

Copy link
Copy Markdown
Collaborator

Is # syntax = still needed?

@tiborvass

tiborvass commented Aug 17, 2018

Copy link
Copy Markdown
CollaboratorAuthor

@thaJeztah you're using the runmount flavored dockefile frontend, instead of secrets.

Note that --secret is now guarded by API version 1.39.

@cpuguy83

Weird that it leaves an empty file in the container image where you mount a secret. Is this an issue with the frontend or with buildkit?

Agreed it's weird, will debug it but shouldn't be a blocker for this PR. I added a couple of tests.

@AkihiroSuda

Is # syntax = still needed?

Yes, this is not part of the stable compiled-in default frontend.

PTAL :)

This patch implements `docker build --secret id=mysecret,src=/secret/file`
for buildkit frontends that request the mysecret secret.
It is currently implemented in the tonistiigi/dockerfile:secrets20180808
frontend via RUN --mount=type=secret,id=mysecret
Signed-off-by: Tibor Vass <tibor@docker.com>
vendors github.com/docker/docker to a7ff19d69a90dfe152abd146221c8b9b46a0903d
Signed-off-by: Tibor Vass <tibor@docker.com>
@thaJeztah

Copy link
Copy Markdown
Member

Tried this again, and looks good :)

printf"hello secret"> ./mysecret.txt
export DOCKER_BUILDKIT=1
docker build --no-cache --progress=plain --secret id=mysecret,src=$(pwd)/mysecret.txt -f - .<<EOF# syntax = tonistiigi/dockerfile:secrets20180808FROM busyboxRUN --mount=type=secret,id=mysecret cat /run/secrets/mysecretRUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobarEOF
#8 [2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
#8 digest: sha256:ab7659590ba7fcf4ec338708fcf824ea76ee7b48b13dce4b9c01faf19395c39d
#8 name: "[2/3] RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret"
#8 started: 2018-08-17 16:08:28.0910424 +0000 UTC
#8 0.303 hello secret#8 completed: 2018-08-17 16:08:28.6508443 +0000 UTC
#8 duration: 559.8019ms
#8 0.303 hello secret
#9 [3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar
#9 digest: sha256:67234c5c8ff0bbd7d6c342980f66f9e5ac75405d2860bbd15747e44291b5cb1d
#9 name: "[3/3] RUN --mount=type=secret,id=mysecret,dst=/foobar cat /foobar"
#9 started: 2018-08-17 16:08:28.6597585 +0000 UTC
#9 0.343 hello secret#9 completed: 2018-08-17 16:08:29.2602578 +0000 UTC
#9 duration: 600.4993ms
#9 0.343 hello secret

@thaJeztahthaJeztah left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@thaJeztah

Copy link
Copy Markdown
Member

/cc @albers for bash completion 😅

@thaJeztah
thaJeztah merged commit cb142fa into docker:masterAug 17, 2018
@GordonTheTurtleGordonTheTurtle added this to the 18.09.0 milestone Aug 17, 2018
@tiborvass

Copy link
Copy Markdown
CollaboratorAuthor

Big thanks to y'all for the review and to Tonis for the implementation!

chrishiestand added a commit to chrishiestand/docker-image-resource that referenced this pull request Aug 22, 2018
It is inaccurate to say that build args will not persist in the final image. They are visible with `docker history` and `docker inspect`.
As soon as possible, we should upgrade to docker 18.09 to support `--secret` - a way to securely pass credentials into the build context.
docker/cli#1288
@haizaar

Copy link
Copy Markdown

@thaJeztah
Can you please elaborate on export DOCKER_BUILDKIT=1?
Will we have to define this env var when this feature becomes available as part of docker-ce 18.09?

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants

@tiborvass@codecov-io@thaJeztah@cpuguy83@AkihiroSuda@haizaar@vdemeester@GordonTheTurtle