Convert Plaster into a DSL

[bwright86]

Not sure if this has been already discussed, but Plaster could benefit from becoming a DSL in Powershell.

Some of the benefits from doing this would be:

• PlasterManifest.xml could become *.Plaster.ps1 (head nod to Pester, PSDeploy, psake, and other modules)
• The manifest Content section could then use foreach loops to create multiple files from array input. (Since this is now a .ps1 file.) (It would resolve Ability to loop through multi-choice parameter #332, Accept collections in plaster parameters  #274, and i think New XML Parameter type - Array #312)
• Readability of the manifest file, and creation of the manifest becomes simpler.
• Powershell's Data Sections allows you to scope specific cmdlets to be used in sections. Making the DSL magic happen. (Link: https://technet.microsoft.com/en-us/library/dd347678.aspx)

The XML nodes in the PlasterManifest.xml are already mostly DSL words, here is an example manifest:

Manifest "Plaster Manifest Template" {
Metadata {
Name "NewPlasterTemplate"
ID "bcd7b44b-0fda-4a08-9857-3e064abd6e2b"
Version 0.0.1
Title "A new Plaster Manifest"
Description "A short description"
Author "Brent W"
Tags Tag1, Tag2, Tag3, etc...
}
Parameters {
Text "ModuleName" "Enter the name of the module"
Choices "License" "Select a license for your module" "Apache" {
Choice '&Apache' "Adds an Apache license file." "Apache"
Choice '&MIT' "Adds an MIT license file." "MIT"
Choice '&None' "No license specified." "None"
}
MultiChoice 'Options' 'Select desired options' @("Pester", "PSake","Git") {
Choice '&Pester test support' "Adds Tests directory and a starter Pester Tests file." "Pester"
Choice 'P&Sake build script' "Adds a PSake build script that generates the module directory for publishing to the PSGallery." "PSake"
Choice '&Git' "Adds a .gitignore file." "Git"
Choice '&None' "No options specified." "None"
}
MultiChoice 'FruitSelection' 'Please pick some fruit' @("Apple", "Banana") {
Choice '&Apple' "Pick an Apple" "Apple"
Choice '&Banana' "Pick a Banana" "Banana"
Choice '&Grapes' "Pick a cluster of grapes" "Grapes"
}
Other 'FullName' 'Enter your full name' 'user-fullname' }
Content {
File 'ReleaseNotes.md' ''
Directory "src\bin"
File 'Tests\*.tests.ps1' 'test\' {
$PLASTER_PARAM_Options -contains "Pester"
}
TemplateFile 'en-US\about_Module.help.txt' 'en-US\about_${PLASTER_PARAM_ModuleName}.help.txt'
foreach ($fruit in $PLASTER_PARAM_FruitSelection) {
TemplateFile '$fruit.json.txt' 'Fruits\$fruit.json'
}
}

Kevin Marquette wrote some good articles on creating a DSL:
https://kevinmarquette.github.io/2017-02-26-Powershell-DSL-intro-to-domain-specific-languages-part-1/

I think the Data Sections could help scope cmdlets as well, not sure if it could replace the constrained runspaces (It would require all scaffolded files to become DSL formatted as well, which is a big ask.)

[TylerLeonhardt]

This is very interesting. I like the concept. I'm wondering what @rjmholt thinks, he's our DSL expert 😉

[rkeithhill]

We went back and forth on this early on and decided on XML because we were sure we could constrain execution using this declarative approach. It wasn't clear to me that a DSL could be constrained. Perhaps that has changed or I was misinformed. Probably the latter.

The other thing we liked about XML was the schema and being able to version and easily validate manifest files against the schema.

When I look at this:

Manifest "Plaster Manifest Template" {
Metadata {
Name "NewPlasterTemplate"
ID "bcd7b44b-0fda-4a08-9857-3e064abd6e2b"
Version 0.0.1
Title "A new Plaster Manifest"
Description "A short description"
Author "Brent W"
Tags Tag1, Tag2, Tag3, etc...
}
Parameters {
...

I don't see much benefit/diff over XML or JSON if you're limiting the manifest to just declarative information without the ability to run arbitrary script. If we decided to punt on constrained execution then a DSL makes more sense IMO.

[bwright86]

I would agree, if the plan is to stick with only declarative information in the manifest, there would be no real benefit to converting from XML to DSL. On the flip side if the decision is to open up the tool, and allow scripting in the manifest and more familiar scripting in the TemplateFiles, there would be plenty of benefits from changing, including:

• Readability/Simplicity (Fueling adoption and lessening the learning curve)
• Extensibility (as people find different use cases for scaffolding)
• Tooling (There will be a need to do more things that are not in scope for this tool, and the community could provide it)

I believe it is better to build powerful tools, and provide useful information to educate the user on how to be safe, as well as tooling to inform the user of the quality/safety of community built templates. It would be helpful to have features that can be enabled as the user advances in their knowledge of the tool. So they can start out with a safe area to experiment and learn and do some basic scaffolding, but as the need to build more advanced templates, they can take on the responsibility of safely scaffolding their templates.

In the end, Plaster is not much different than Powershell, user's can write scripts of varying quality/safety, and they can execute it in their environment. They also have the ability to download scripts from the internet, and it is up to them to decide on the quality/safety of the script before executing it. I believe for Plaster to continue to grow as a scaffolding tool into a fully grown scaffolding system, it will be necessary to move away from a safety-first mentality.

Thank you for the response.

[rjmholt]

I went to @KevinMarquette's talk at PowerShell Summit actually. It was really interesting and informative.

The hard part I think is limiting the execution. I'm not really sure how you'd go about fixing that part in current PowerShell. That is an interesting concern though, which is helpful because I keep thinking about how to improve DSL support in PowerShell.

Even using type constraints and a hashtable body, you still don't prevent arbitrary execution - the RHS of hashtable expressions can be any expression at all, and type constraints will only be applied after the evaluation of an expression.

The plus side is that you control when the scriptblock executes. So if you can find a way to check the scriptblock for bad things, you can do that check and throw an error if it does. Problem is that that checking is hard and not terribly universal -- you would have to spelunk the AST and do some heuristics, which there are helper methods for, but still. Example:

$dslBody= {
Invoke-BadCommand
}
$exprAsts=$dslBody.Ast.Find({ $args[0] -is [System.Management.Automation.Language.ExpressionAst] },$true)
Test-BodyExpressionsAreSafe$exprAsts

A particularly helpful method in that case is $ast.SafeGetValue(), which tries to get a value from the AST node if it is safe to do so.

Anyway, even in a world with an XML/JSON-based configuration, you can use this quite easily -- you just write a DSL that constructs the XML (this is a pretty rough example, you would want one function for each keyword, and then functions that handle each layer of the XML internally):

functionManifest
{
param([hashtable]$Body)
if ($Body.Metadata)
{
$metadata= [System.Xml.Linq.XElement]::new([System.Xml.Linq.XName]"metadata")
$metadataProperties=@{
name= [string]
id= [guid]
version= [version]
title= [string]
description= [string]
author= [string]
tags= [string[]]
}
foreach ($metadataPropertyin$metadataProperties.Keys)
{
if ($Body.Metadata[$metadataProperty])
{
$val= [System.Management.Automation.LanguagePrimitives]::ConvertTo($Body.Metadata[$metadataProperty],$metadataProperties[$metadataProperty])
$metadata.Add([System.Xml.Linq.XElement]::new([System.Xml.Linq.XName]$metadataProperty,$val))
}
}
}
$doc= [System.Xml.Linq.XElement]::new([System.Xml.Linq.XName]"plasterManifest",$metadata)
return$doc.ToString()
}

This gives you something like this:

> Manifest @{
>>Metadata=@{
>>name="Test Module">>id= [guid]::NewGuid()
>>description="Something I wrote">> }
>> }
<plasterManifest><metadata><name>Test Module</name><description>Something I wrote</description><id>cc62e080-7788-4f41-8406-efa4b4f1715e</id></metadata></plasterManifest>
C:\Users\roholt\Documents\Dev\sandbox
>

With this model, scriptblocks essentially refer to arrays or unordered bags with possible duplicate keys. So for <parameter/>s, you just stick them in a script block and call & on them.

[KevinMarquette]

Just to add to this conversation, there is a community module that already implements this as a DSL. https://github.com/dchristian3188/PlasterManifestDSL@dchristian3188

[rkeithhill]

I remember a few years back Jason gave a session on DSL ideas/improvements during the MVP summit. I had suggested that we needed some mechanism to restrict what the DSL could execute.

I think the DSL issue is somewhat orthogonal to the "allow arbitrary/extended execution" request. I think the latter can be solved with an adequately named param like -AllowArbitraryScriptExecution.

[bwright86]

Thanks @KevinMarquette and @rjmholt for the input, I did not realize there is already a module that adds DSL to Plaster. This would probably work for my needs.

For this issue, I was looking for a way to extend the Content section of the manifest to allow looping of multiple templates based on the some input.

Something like this:

Content {
foreach ($fruit in $PLASTER_PARAM_FruitSelection) {
TemplateFile '$fruit.json.txt' 'Fruits\$fruit.json'
}
}

But the discussion did lean into #333, where I asked to open up the constrained runspace for more advanced use cases. That issue does have some overlap here, but if something else is creating the manifest without a constrained runspace, I think that would resolve this.

I will try out the module Kevin M. pointed out, and see if that covers what I was looking for.
I could be good to close this issue, unless there is more to discuss.

Thanks again for the responses.

[TylerLeonhardt]

I think it would be worthwhile to include a passage in the readme pointing to @dchristian3188's PlasterManifestDSL in case others are looking for a DSL.

And to talk about how we can make sure that repo keeps up-to-date with any breaking schema changes in Plaster.

:)

[torgro]

Hi, a bit late to the party, however regarding the arbitrary command execution problem for DSL, would not the scriptblock method CheckRestrictedLanguage solve the issue? The method takes 3 arguments:

• An IEnummerable collection of allowed variables, typically an array or List.
• An IEnummerable collection of allowed commands, typically an array or List.
• A boolean value that indicates if use of Environment variables is allowed ($true makes them allowed)

/Tore

[jdhitsolutions]

Here is additional information related to this concept of an re-architected Plaster. #366 (comment)

[JustinGrote]

My two cents is that having a transpiler DSL to generate the Plaster XML makes sense, in the same way Bicep is used to generate ARM templates in a friendlier way while maintaining maximum compatibility.

Working with the Plaster XML directly is painful even with @code tooling, etc. and ideally I think it should switch to JSON as the manifest format but because the XML serialization is very bespoke today that would be difficult. Ideally the plaster manifest configuration would just be a C# type/s with all the appropriate json and xml attributes, and then it could be imported/exported to json/yaml/toml/whatever with the appropriate serializer.

The constrained runtime environment issue is real, all it takes is a "trusted" template to suddenly get a push that exploits a user's computer for Plaster to immediately become persona-non-grata. I know this is true of Powershell Modules too but still. I feel the hard work already done in plaster has been done, and this also wouldn't break compatibility for existing templates.

The advantage of a Transpiler DSL too is that it could have a "plugin" to editor services for best practices as well as continuously compile the manifest and report issues like Bicep does.

[JustinGrote]

If instead Plaster was being done as a "ground-up" rethinking, my vote would be for leveraging the dotnet templating engine, consuming those types, and not reinventing the wheel there would be the ideal approach, since Powershell pins its lifecycle to dotnet anyways, and the assemblies are pretty portable and could be included in the plaster module.

The many other templating engines are going to either require a large external dependency (e.g. yeoman would require Pester to tag along or install all of NodeJS), or are more risky dependency that may be abandoned.

[JustinGrote]

On this subject, I started a new proof of concept project Mortar and the design doc is up.
https://github.com/JustinGrote/Mortar

[jasonchester]

ModuleBuilder has a dotnet new template for scaffolding new module projects.

https://github.com/PoshCode/ModuleBuilder/blob/master/Templates/README.md

[JustinGrote]

That's the same idea here, with a simpler process and not requiring dotnet.exe to be installed eventually.