Load middleware and phases from middleware.json

[bajtos]

Sample JSON:

 {
"routes:before": {
"morgan": {
"params": ["dev"]
}
},
"routes": {
"loopback/server/middleware/rest": {
}
},
"subapps": {
"./adminer": {
},
}
}

The JSON file can be customized using the usual conventions:

• middleware.local.{js|json}
• middleware.{env}.{js|json}

It is also possible to mount the same middleware in the same phase
multiple times with different configuration.

Example config:

{
"auth": {
"oauth2": [
{
"params": "first"
},
{
"params": "second"
}
]
},
});

A part of #44

/to @raymondfeng please review
/cc @ritch

[bajtos]

In the light of our recent discussion in SlackChat and the request strongloop/loopback#794, we cannot use middleware path as a key, as it must be possible to load multiple instances of the same middleware.

I'll rework this pull request to support that.

[ritch]

factoryFile is a bit weird... I'm not 100% against it... but it just doesn't look right. Perhaps middlewareFile?

[bajtos]

How about sourceFile? It is already nested in middleware section, I feel it's not necessary to repeat middleware in the name.

[ritch]

sourceFile works

[bajtos]

Renamed.

[ritch]

@bajtos I took a look and didn't find anything worth mentioning besides factoryFile. 👍

[bajtos]

I have reworked the patch to support strongloop/loopback#794, the file middleware.json has a new layout now.

• First-level keys define phase names. We are leveraging the fact that JSON deserialization preserves the order of keys, thus the order of first-level keys defines the order of phases.
• At the second level, the key is the path to the middleware file. This makes it easy to reference any particular middleware instance in middleware.production.json and friends, since the key is phase+middleware instead of a positional index in an array of all midleware.
• The value is either an object describing a single middleware instance to create/mount, or an array of objects if you want to configure multiple instances. This is the only situation when you have to depend on array indexes in middleware.production.json.

Example:

{
"auth": {
"oauth2": [
{
"args": "first-domain-config"
},
{
"args": "second-domain-config"
}
]
},
"routes:before": {
"morgan": {
"args": ["dev", { "threshold": 128 }]
}
},
"routes": {
"loopback/server/middleware/rest": {
}
},
"subapps": {
"./adminer": {
},
}
}

Once we have strongloop/loopback#794 implemented, we can use a config similar to the following example to configure OAuth2 protection for different paths:

{
"auth": {
"loopback-component-oauth2/server/middleware/authorize": [
{
"paths": ["/restricted", "/api"],
"args": { "session": false, "scope": "demo"}
},
{
"paths": ["/admin"],
"args": { "session": true, "scope": "intranet" }
},
]
}
}

@ritch please review the new version
@raymondfeng please take a look too if the solution makes sense in the context of loopback-component-oauth2

[bajtos]

The new version depends on loopback changes proposed in strongloop/loopback#796.

[raymondfeng]

@bajtos I'm trying to figure out what is in the oauth2 component and what is in the main app.

For the oauth2 component, I assume we will implement (and declare) the middleware. Let's take the token middleware (validates the access token and stores the accessToken object in the req and/or context) as an example.

• loopback-component-oauth2/server/middleware/token.js (The js file that exports a function to create token middleware using options)
• loopback-component-oauth2/server/middleware/token.json (The optional json file that describes the middleware, including the model for config and default values).

In the main app,

• server/middleware-config.json (The json file as you proposed to configure middleware and attach them to one of the phases).

Can you confirm?

[ritch]

Can we hide the structure of the component somehow by handling this a bit differently:

loopback/server/middleware/rest

Perhaps reference this as rest? or loopback.rest? I don't really care what the reference is as long as "server/middleware" isn't repeated in all middleware.json files.

[ritch]

args is weird... It hints at a function call which is a bit odd in a config file. Although I'm not 100% against it since we get the same behavior.

[ritch]

@bajtos do you have ideas on how to implement this in workspace / the generators? The format in general is nice / readable, but looks like it will be difficult to implement in the workspace.

[raymondfeng]

Let's walk through the developer experience.

For the component developer that implements the middleware

1. Write a function as the factory to create middleware handlers, for example,
   foo-middleware.js

module.exports=function(options){returnfunctionfooMiddleware(req,res,next){// foo logic ...next();}}

1. Optionally make it exported via the module main script, for example,
   index.js

exports.foo=require('./lib/foo-middleware.js');

1. Document the middleware in README.md, such as:
   The my-component module exports a foo middleware, it can be used as follows:

varfoo=require('my-component').foo;app.use(foo(options));

• Options for foo
  • ttl: Number time-to-live in seconds for cache
  • host: The redis server to reload the cache

For the application developer uses the middleware from my-component.

1. npm install my-component
2. Read the docs to understand which middleware is available and what configuration is needed
3. Write code to use foo:

varoptions={ttl: 60};// or read from a config filevarfoo=require('my-component').foo;app.use(foo(options));

@bajtos Your proposal seems to automate step 3 for the application (importing the middleware from my-component). We can start from there. What's your take on the component side (exporting the middleware)?

[bajtos]

@raymondfeng@ritch thank you for thoughtful comments!

Just to make it clear: the first version of middleware support will not include tooling (slc loopback, strong-studio). Therefore things like loopback-component-oauth2/server/middleware/token.json (The optional json file that describes the middleware, including the model for config and default values) is out of scope of this work.

However, it is still a good idea to discuss the tooling to ensure that it will be possible to implement the necessary changes in the future.

Keeping that in mind, here are my answers to the questions raised above.

@raymondfeng
For the oauth2 component, I assume we will implement (and declare) the middleware. Let's take the token middleware (validates the access token and stores the accessToken object in the req and/or context) as an example.

• loopback-component-oauth2/server/middleware/token.js (The js file that exports a function to create token middleware using options)
• loopback-component-oauth2/server/middleware/token.json (The optional json file that describes the middleware, including the model for config and default values).

As I am envisioning the final solution, the runtime (loopback-boot) does not need any middleware declaration, as it has all necessary information in server/middleware.json. Any declaration and metadata (e.g. a model for config and default values) will be used solely by slc loopback and strong-studio to build user-interface for installing and configuring the middleware in the project.

In the main app,

• server/middleware-config.json (The json file as you proposed to configure middleware and attach them to one of the phases).

The current name is server/middleware.json, it is similar to server/datasources.json.

Middleware exported by a module (component) should be described in package.json together with other component metadata, therefore it should be ok to use the file name middleware for configuration of middleware that is used by the app.

However, if both of you and @ritch prefer middleware-config.json over middleware.json, then I am ok with changing the file name.

For the component developer that implements the middleware (etc.)

Yes, your description is correct. In the future, we should automate some part of that process via slc loopback.

For the application developer uses the middleware from my-component.

1. npm install my-component
2. Read the docs to understand which middleware is available and what configuration is needed
3. Write code to use foo:
   js var options = {ttl: 60}; // or read from a config file var foo = require('my-component').foo; app.use(foo(options));
   Your proposal seems to automate step 3 for the application (importing the middleware from my-component). We can start from there.

Yes. Additionally, my proposal handles the following:

• mounting the middleware at the correct position in the middleware chain via phases
• allowing tools to add and modify middleware entries programatically, as they are described in JSON and not in javascript

What's your take on the component side (exporting the middleware)?

For the initial version, the component does not have to do anything to export a middleware. Just put the javascript file to server/middleware/{name}.js and describe it in component documentation.

In the future, the component should include middleware description in the component metadata specified in package.json, e.g.

{"name": "loopback-component-foo","version": "1.0.0","loopback-component": {"common": {/* ... */},"server": {"connectors": /*...*/,/* models? */"middleware": {"token": {"description": "The token middleware validates the access token and stores the accessToken object in the req and/or context","sourceFile": "server/middleware/token","recommendedPhase": "auth","options": {"ttl": {"type": "number","default": 60,"description": "Time to live for the tokens."}}}}}}}

Storing the component metadata in package.json makes it possible to fetch component metadata directly from the npmjs registry without having to install the npm package.

@ritch
Can we hide the structure of the component somehow by handling this a bit differently:

loopback/server/middleware/rest

Perhaps reference this as rest? or loopback.rest? I don't really care what the reference is as long as "server/middleware" isn't repeated in all middleware.json files.

Definitely not rest:

• As we have already learned with models, using one global namespace scales very poorly. Consider token from loopback and loopback-component-auth2.
• The current design supports existing express middleware where the module is the factory function. By using the middleware file name as the id, we may get a conflict between loopback/server/middleware/foobar and node_modules/foobar.

So far, loopback-boot is trying to avoid any opinions about the layout of 3rd party modules, for example _meta.sources in model-config.json can point to any directory.

I would like to preserve that trait for the middleware registration too and keep support for "require()"-style identifiers in addition to any shorthand convention-based notation we can come up with.

Let's move the discussion about the shorthand notation to a new github issue #68 and implement the feature as an incremental addition once this PR is landed.

Do you have ideas on how to implement this in workspace / the generators? The format in general is nice / readable, but looks like it will be difficult to implement in the workspace.

The format is an implementation detail of the filesystem storage, it does not have to drive the design of workspace entities.

Here is a quick outline of a possible implementation in loopback-workspace:

• Facet.middlewarePhases: [string] - an ordered list of phase names

• FacetMiddlewareConfig - the entity representing a single entry in middleware.json

  {
  "id": {"type": "string", "id": true, "json": false},
  "facetName": {"type": "string", "required": true, "json": false},
  "sourceFile": {"type": "string", "required": true},
  "phase": {"type": "string", "required": true},
  "enabled": {"type": "boolean", "default": true},
  "args or params": {"type": "array"},
  "paths": { "type": "[string]", "description": "TBD - see loopback#794 Scope middleware to a subpath" }
  }

• FacetMiddlewareConfig.phase must reference an existing phase in Facet.middlewarePhase

• We need to implement a custom function for loading and saving middleware.json. We have already done something similar with model-config.json that represents ModelConfig[] and Facet.modelsMetadata.

[bajtos]

@ritch args is weird... It hints at a function call which is a bit odd in a config file. Although I'm not 100% against it since we get the same behavior.

Let's move this discussion to strongloop/loopback#796

[bajtos]

For the application developer uses the middleware from my-component.

1. npm install my-component
2. Read the docs to understand which middleware is available and what configuration is needed

This should be automated by slc loopback:install my-component in the future, but not as part of this PR.

[bajtos]

@raymondfeng@ritch any more comments? I need to finish this patch before I can start working on follow-up stories scheduled for Sprint 58.

[raymondfeng]

@bajtos Can you confirm if the following middleware.json can be loaded by your PR?

https://gist.github.com/raymondfeng/25f3dc5acd3ce60dc98f

I have a few things around the middleware key. In my case, I have:

session ==> loopback.session
./middleware/proxy ==> The relative path (against middleware.json) for require

[bajtos]

@bajtos Can you confirm if the following middleware.json can be loaded by your PR?
https://gist.github.com/raymondfeng/25f3dc5acd3ce60dc98f

It looks like it should be loadable, possibly with minor tweaks in the syntax.

I have a few things around the middleware key. In my case, I have:
session ==> loopback.session

No, this won't be implemented, as session may clash with a module name in node_modules. See my comment above:

• As we have already learned with models, using one global namespace scales very poorly. Consider token from loopback and loopback-component-auth2.
• The current design supports existing express middleware where the module is the factory function. By using the middleware file name as the id, we may get a conflict between loopback/server/middleware/foobar and node_modules/foobar.
• So far, loopback-boot is trying to avoid any opinions about the layout of 3rd party modules, for example _meta.sources in model-config.json can point to any directory.

I would like to preserve that trait for the middleware registration too and keep support for "require()"-style identifiers in addition to any shorthand convention-based notation we can come up with.

Let's move the discussion about the shorthand notation to a new github issue #68 and implement the feature as an incremental addition once this PR is landed.

#68 proposes loopback/session. IMHO including loopback/ prefix as a namespace is a good thing, as it makes it clear where is the middleware coming from.

./middleware/proxy ==> The relative path (against middleware.json) for require

Yes, this will work. The path is relative to appRootDir at the moment, but since middleware.json is stored in appRootDir, the result is the same.

[raymondfeng]

Sounds good. For the relative path, I would prefer to be resolved against the containing file as it won't leave any ambiguity to developers to specify the path.

[ritch]

👍

[bajtos]

Sounds good. For the relative path, I would prefer to be resolved against the containing file as it won't leave any ambiguity to developers to specify the path.

For posterity, I have amended the code to make this more clear and to guide future developers in that direction.

// not configurable yetvarmiddlewareRootDir=appRootDir;varmiddlewareConfig=options.middleware||ConfigLoader.loadMiddleware(middlewareRootDir,env);varmiddlewareInstructions=buildMiddlewareInstructions(middlewareRootDir,middlewareConfig);

[bajtos]

@slnode test please