475 Pattern Matching Callbacks (codenamed wildcards)

[alexcjohnson]

Pattern Matching Callbacks! (Codenamed, Wildcards!)

➡️ See official documentation here: https://dash.plotly.com/pattern-matching-callbacks

Closes#475 - using the API discussed there (I'll post some examples soon, but for now here's the description):

• String IDs still work the same way in both layout and callbacks
• Now you can also make dict IDs. Keys should be strings, values can be strings, numbers, booleans, like:
  • id={"type": "cat", "age": 4, "purrs": True}
  • IDs must still be unique
• Callbacks can use dict IDs with wildcard values for any of the keys. The supported wildcards are MATCH, ALL, and ALLSMALLER - and can be imported from dash.dependencies. There are various rules about how these can combine, driven by the need to resolve these to a unique callback invocation for each set of outputs and identify uniquely the inputs required for this invocation...

To do before this is ready:

• Move most callback validation logic to the front end (so it doesn't need to be replicated in R)
• Do something with the callback graph when there are wildcards
• Test that this PR closes callback failure bugs: [BUG] Callback with canceled update intermittently prevents later callback #1053, [BUG] UI components not updating from dash_table connected callback #1071, [BUG] Multi-output leads to callback results being ignored #1084, [BUG] Data Table prevents other callbacks until it is rendered first #1105
• See if it also closes redundant callback bugs Callbacks called multiple times unnecessarily #635, [BUG] callbacks called in wrong order / too often #832

Contributor Checklist

• I have run the tests locally and they passed. (refer to testing section in contributing)
• I have added tests, or extended existing tests, to cover any new features or bugs fixed in this PR
• I have added entry in the CHANGELOG.md
• If this PR needs a follow-up in dash docs, community thread, I have mentioned the relevant URLS as follow
  • this github #PR number updates the dash docs
  • here is the show and tell thread in plotly dash community

Closes#832 (among other callback bugs - the others all have commits that close them with matching tests, but #832 is a bit heavy for a test of its own)
Closes#1146

[christianwengert]

Dear @alexcjohnson
This looks promising!
If you deem it useful, I might start some tests for my usecases next week.

[alexcjohnson]

@christianwengert Absolutely! Thank you for your patience with this. I will try to post some usage examples here later today.

[christianwengert]

Yes, an usage example would be great for getting started

…

On 31 Jan 2020, at 15:44, alexcjohnson ***@***.***> wrote: @christianwengert Absolutely! Thank you for your patience with this. I will try to post some usage examples here later today. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[alexcjohnson]

To-Do list example

The edit_list callback creates pairs of elements: a checkbox for marking items done and a div displaying the item text. Both of these have dict IDs, with matching item values, but the checkbox is distinguished by the "action": "done" entry. The callbacks then refer to these items with "item": ALL or "item": MATCH in their input/output/state wildcard component IDs.

importdashimportdash_core_componentsasdccimportdash_html_componentsashtmlfromdash.dependenciesimportInput, Output, State, MATCH, ALLapp=dash.Dash(__name__)
app.layout=html.Div([
html.Div('Dash To-Do list'),
dcc.Input(id="new-item"),
html.Button("Add", id="add"),
html.Button("Clear Done", id="clear-done"),
html.Div(id="list-container"),
html.Hr(),
html.Div(id="totals")
])
style_todo= {"display": "inline", "margin": "10px"}
style_done= {"textDecoration": "line-through", "color": "#888"}
style_done.update(style_todo)
@app.callback( [Output("list-container", "children"),Output("new-item", "value") ], [Input("add", "n_clicks"),Input("new-item", "n_submit"),Input("clear-done", "n_clicks") ], [State("new-item", "value"),State({"item": ALL}, "children"),State({"item": ALL, "action": "done"}, "value") ])defedit_list(add, add2, clear, new_item, items, items_done):
triggered= [t["prop_id"] fortindash.callback_context.triggered]
adding=len([1foriintriggeredifiin ("add.n_clicks", "new-item.n_submit")])
clearing=len([1foriintriggeredifi=="clear-done.n_clicks"])
new_spec= [
(text, done) fortext, doneinzip(items, items_done)
ifnot (clearinganddone)
]
ifadding:
new_spec.append((new_item, []))
new_list= [
html.Div([
dcc.Checklist(
id={"item": i, "action": "done"},
options=[{"label": "", "value": "done"}],
value=done,
style={"display": "inline"}
),
html.Div(text, id={"item": i}, style=style_doneifdoneelsestyle_todo)
], style={"clear": "both"})
fori, (text, done) inenumerate(new_spec)
]
return [new_list, ""ifaddingelsenew_item]
@app.callback(Output({"item": MATCH}, "style"), [Input({"item": MATCH, "action": "done"}, "value")])defmark_done(done):
returnstyle_doneifdoneelsestyle_todo@app.callback(Output("totals", "children"), [Input({"item": ALL, "action": "done"}, "value")])defshow_totals(done):
count_all=len(done)
count_done=len([dfordindoneifd])
result="{} of {} items completed".format(count_done, count_all)
ifcount_all:
result+=" - {}%".format(int(100*count_done/count_all))
returnresultif__name__=="__main__":
app.run_server(debug=True)

[alexcjohnson]

I noticed a problem with the To-Do app: if you delete all the items, the totals callback will not fire to say "0 of 0 items completed" as it should. This is because we currently only fire callbacks based on inputs that exist when they're fired, but for ALL (and ALLSMALLER) an input being removed is itself a change to the input. In fact this could happen when removing just some of the items, but in this particular case the callback still fires because we replace all the other list items (having not yet implemented #968 array operation callbacks).

Working on a fix for this...

[alexcjohnson]

Extended To-Do app using ALLSMALLER to link up the list items in sequence:

ALLSMALLER can only be used in Input and State items, and must be used on a key that has MATCH in the Output item(s).

importdashimportdash_core_componentsasdccimportdash_html_componentsashtmlfromdash.dependenciesimportInput, Output, State, MATCH, ALL, ALLSMALLERapp=dash.Dash(__name__)
app.layout=html.Div([
html.Div('Dash To-Do list'),
dcc.Input(id="new-item"),
html.Button("Add", id="add"),
html.Button("Clear Done", id="clear-done"),
html.Div(id="list-container"),
html.Hr(),
html.Div(id="totals")
])
style_todo= {"display": "inline", "margin": "10px"}
style_done= {"textDecoration": "line-through", "color": "#888"}
style_done.update(style_todo)
@app.callback( [Output("list-container", "children"),Output("new-item", "value") ], [Input("add", "n_clicks"),Input("new-item", "n_submit"),Input("clear-done", "n_clicks") ], [State("new-item", "value"),State({"item": ALL}, "children"),State({"item": ALL, "action": "done"}, "value") ])defedit_list(add, add2, clear, new_item, items, items_done):
triggered= [t["prop_id"] fortindash.callback_context.triggered]
adding=len([1foriintriggeredifiin ("add.n_clicks", "new-item.n_submit")])
clearing=len([1foriintriggeredifi=="clear-done.n_clicks"])
new_spec= [
(text, done) fortext, doneinzip(items, items_done)
ifnot (clearinganddone)
]
ifadding:
new_spec.append((new_item, []))
new_list= [
html.Div([
dcc.Checklist(
id={"item": i, "action": "done"},
options=[{"label": "", "value": "done"}],
value=done,
style={"display": "inline"}
),
html.Div(text, id={"item": i}, style=style_doneifdoneelsestyle_todo),
html.Div(id={"item": i, "preceding": True}, style=style_todo)
], style={"clear": "both"})
fori, (text, done) inenumerate(new_spec)
]
return [new_list, ""ifaddingelsenew_item]
@app.callback(Output({"item": MATCH}, "style"), [Input({"item": MATCH, "action": "done"}, "value")])defmark_done(done):
returnstyle_doneifdoneelsestyle_todo@app.callback(Output({"item": MATCH, "preceding": True}, "children"), [Input({"item": ALLSMALLER, "action": "done"}, "value"),Input({"item": MATCH, "action": "done"}, "value") ])defshow_preceding(done_before, this_done):
ifthis_done:
return""all_before=len(done_before)
done_before=len([1fordindone_beforeifd])
out="{} of {} preceding items are done".format(done_before, all_before)
ifall_before==done_before:
out+=" DO THIS NEXT!"returnout@app.callback(Output("totals", "children"), [Input({"item": ALL, "action": "done"}, "value")])defshow_totals(done):
count_all=len(done)
count_done=len([dfordindoneifd])
result="{} of {} items completed".format(count_done, count_all)
ifcount_all:
result+=" - {}%".format(int(100*count_done/count_all))
returnresultif__name__=="__main__":
app.run_server(debug=True)

[alexcjohnson]

At a meeting where I demoed this feature, the consensus was ALL and ANY are easy to confuse with each other (and in a separate discussion @christianwengert seems to agree) - The proposal I think we agreed on is to change ANY to MATCH - which makes it clear that the values will match across Input and Output items for a given callback invocation.

We didn't discuss ALLSMALLER, as I didn't have a demo ready for it yet. Is it clear from the extended To-Do demo above what this does and how to use it?

Give me a 👍 reaction (not a comment please) if the names ALL, MATCH, and ALLSMALLER look good to you. If not and you have another suggestion please comment.

[nicolaskruchten]

ALLSMALLER seems oddly specific to me... why not also ALLBIGGER ?

[Marc-Andre-Rivet]

💃💃💃 🚢 ⛵️ 🎉🎉🎉

[alexcjohnson]

The issue @chriddyp mentioned #1103 (comment) from the docs, that the title never goes back from "Updating...", is still present. Investigating...

[alexcjohnson]

Ah got it - we're creating and throwing some errors that I forgot to trap, so I could clear the associated callback from the queue and surface the error in the devtools.

The specific error we're hitting in the docs is (in case someone wants to fix it):

Uncaught (in promise) ReferenceError: A nonexistent object was used in an Input of a Dash callback. The id of this object is memory-button and the property is n_clicks. The string ids in the current layout are: [memory, memory-output, local, session, wait-for-layout, location, banner, logo-home, backlinks-top, chapter, backlinks-bottom, pagemenu]

[alexcjohnson]

After 7c47b05 (running docs with devtools on so we see the errors... with devtools off the title is still correct and the errors only appear in the js console)

[MM-Lehmann]

whats that rationale behind the restriction that I cannot combine MATCH with other (static) outputs? I would like to have a MATCH callback on some GUI elements which save their current state into a dcc.Store object and retrieve it later from there. Is it because in the background multiple callbacks are created for each match of objects and those can't have duplicate outputs between each other?

[alexcjohnson]

That’s essentially it - the value to return into that single output would be ill-defined, as it would be trying to get a separate value from each matched input.

I see how specifically with Store it would be possible to construct the callback in such a way as to make it self-consistent. But that kind of construction would be tricky and fragile. Keeping MATCH keys the same across all outputs makes this much simpler for everyone to reason about (including us writing the rendering engine!) but in return you sometimes have to handle your data a little differently.

In this case I would suggest either making a separate Store for each MATCH value, or a separate callback that combines the inputs with ALL and returns the single Store value.

[gmountee]

When I used two outputs, one of which uses MATCH and the other has nothing to do with 'index':
@app.callback(
[Output({'type': 'dynamic-output', 'index': MATCH}, 'children'),
Output('title_button', 'children')],
[Input({'type': 'dynamic-dropdown', 'index': MATCH}, 'value')],
[State({'type': 'dynamic-dropdown', 'index': MATCH}, 'id')],
)

it raised an error as follows:

In the callback for output(s):
{"index":MATCH,"type":"dynamic-output"}.children
title_button.children
Output 1 (title_button.children)
does not have MATCH wildcards on the same keys as
Output 0 ({"index":MATCH,"type":"dynamic-output"}.children).
MATCH wildcards must be on the same keys for all Outputs.
ALL wildcards need not match, only MATCH.

Why is it necessary to require all the outputs with MATCH, and how can I solve this problem?
Thanks!!!

[alexcjohnson]

@lhsusan1016 as described in #1103 (comment) if there were two different elements that satisfy your MATCH pattern, then there would need to be two different invocations of this callback, one for each. Then this would be two different values for title_button.children - so which one would we use? It’s ambiguous and therefore an error.

You can solve this by either switching to an ALL pattern to create all the dynamic outputs at once, or making a separate callback for title_button.children

I happened to see your comment, but old closed PRs are not great places to start conversations. This kind of question is better at https://community.plotly.com