Blocks as the lingua franca of try-wordpress

[psrpinto]

As we dove deeper into the development of try-wordpress, I noticed some patterns emerge out of the code that made me look at things through a different lens. I wanted to write down these thoughts to clarify my own understanding and to gather feedback, so here goes.

How try-wordpress works today

Today, try-wordpress works as follows. We take the information from the source site, convert it to an intermediate format (json) and pass it to a custom API endpoint, which will then persist the data wherever it needs to go (e.g. wp_posts):

We have been struggling to define what this intermediate format should look like, which I think is expected because it needs to be able to represent all types of data we'll encounter, so it's not a simple problem to solve.

However, I think there is an alternative approach that would allow us to remove the need for this intermediate format, and bring along additional advantages. The remainder of this post will describe this alternative approach.

High-level overview

If I think of the goal of try-wordpress in a really abstract way, I would describe it as allowing a user to easily author a new WordPress site. What differentiates try-wordpress however, is that the new WordPress site will be authored from a pre-existing source site. Or in other words:

A user traditionally authors a WordPress site by transferring information from their brain into a format that WordPress understands. They do so by using the block editor (or the site editor; from now on, when I mention block editor I mean either the block editor or the site editor):

In try-wordpress, the scenario is very similar, but instead of the information coming from the user's brain, it comes from the source site. So we can think of try-wordpress as a black box that has the same role that the block editor has in the traditional scenario, which would allow us to represent data flow as follows:

This results in there no longer being a need for an intermediate format, and thus neither a custom REST API that transforms the intermediate format to the format that WordPress expects.

This results in a significantly smaller surface where try-wordpress needs to operate in. We don't need to implement anything on the right side of the diagrams above, and we're left with the problem of converting the raw information to blocks + attributes (more on this below).

Additionally, by speaking the same language as the Block editor, try-wordpress would seamlessly integrate into the existing WordPress ecosystem. Because we would make the exact same API calls that the Block editor makes, WordPress would in fact not know or care whether a request came from the block editor or from try-wordpress.

Raw information to blocks

As mentioned above, by making try-wordpress "block-native", we would be left with the problem of converting raw information to blocks + attributes (the left section in the diagrams above).

This is of course a very complex problem, and a solution for such a problem would likely be useful in contexts other than try-wordpress. For this reason, I think it would make sense to extract this responsibility out of try-wordpress into its own thing, maybe as part of WordPress/wordpress-playground#1888.

For the time being, since such a parser does not exist yet, we could implement an ad hoc parser in try-wordpress, with the bare minimum functionality needed for us to implement an MVP of try-wordpress.

Auxiliary data

All that has been written above has to do with user data, i.e. data that comes from the source site. However, for try-wordpress to do its function, it also needs to store additional data that relates to the import process itself. As an example, for each imported blog post, we would need to store:

• The source URL from which it was imported.
• The CSS selectors that were used to extract the content from the original page.
• The original raw HTML, so that we can re-run parsing without needing to crawl the original site again.

This auxiliary data would be persisted through a custom REST API, that we would design to address the specific needs of try-wordpress. This would result in the following data flow:

Continuing on the blog post example from above, for each imported blog post, we would make requests to the Core API and to the Custom REST API:

• Requests relating to user data go to the Core REST API.
• Requests relating to auxiliary data go to the Custom REST API.

Extensibility

One major unknown we have today relates to how to make try-wordpress work with any post type. What does a plugin developer need to do to make try-wordpress support their plugin? For example, if I develop an e-commerce plugin, how can I make it so that a try-wordpress user can import their existing e-commerce site into my plugin?

Another related subject is that we don't want there to be a need for UI development for every post type we need to support. The UI should seamlessly adapt to the post type, so that it shows the fields that are relevant for that post type. For an e-commerce product, this could be a title, a description and a price.

How can we solve both these problems in try-wordpress? If we think of try-wordpress as an "alternative block editor" as described in previous sections, then the answer becomes: We do it in the same way the block editor does. To determine what fields a certain post type has, we would make the same API calls that the block editor makes.

Now, the answer to the question "What does a plugin developer need to do to make try-wordpress support their plugin" is simply: they need to make it work in the block editor, at which point it automatically also works in try-wordpress.

Conclusion

By thinking of try-wordpress as a black box that behaves as the block editor (i.e. it makes the same REST API calls as the block editor), we would be able to massively reduce the surface of where try-wordpress needs to operate. We would remove the need for an intermediate format and for custom API endpoints that deal with user data (we'd still have custom endpoints for auxiliary data).

Additionally, and more importantly, try-wordpress would automatically work with any custom post type, provided that that custom post type works in the block editor.

Next steps

If we end up adapting the solution proposed above, I would imagine the next steps to be:

1. Design the Custom REST API (auxiliary data)
2. Implement the Custom REST API
3. Implement a generic import UI that adapts to the the structure of the post type being imported, so that it automatically shows the fields for that post type.
4. Implement an ad hoc html to block markup parser.
5. Convert the existing implementation of blog post import to this new architecture.
6. Convert the existing implementation of navigation import to this new architecture.
7. Implement crawling (i.e. the feature that automatically imports all posts of a given post type).
8. try-wordpress core features are "done".

[psrpinto]

/cc@ashfame@akirk@adamziel

I welcome any feedback you might have.

[akirk]

What you propose sounds compelling but I am worried about the "where" of where transformations happen and that it's not feasible to implement in practise.

If I understand you correctly, you propose to have the "content detection" be a Gutenberg paste handler so that when you paste in a product, it would create the right blocks that a product has. It could then store the respective data in the blocks it finds. This means that the content detection moves away from the user and into autodetecting it in the paste handler. I am not sure this would work well.

In the current model, we'd create structured data by the user selecting the fields. This would be passed on as that auxiliary data you mention, and stored as post_meta, as well as being provided to the do_action() for plugins to receive.

I like that current model better because it separates out the parsing concerns better. We can work on automatically detecting the right fields on the page using something like https://github.com/fivefilters/ftr-site-config definitions and send a defined set of metadata per defined entity.

If the detection would happen in blocks it means that each plugin would have to implement that data detection and/or we'd have to juggle with data liberation provided blocks and do the content detection there.

[psrpinto]

If I understand you correctly, you propose to have the "content detection" be a Gutenberg paste handler so that when you paste in a product, it would create the right blocks that a product has.

I think maybe I didn't make that part clear. The current model that we have today, where the user clicks on individual fields, would remain the same. The user would not be clicking on the "container" element, they would still click on individual fields.

In the approach suggested here, the difference is that which fields are shown for a given post type, comes from wherever the block editor goes look for that information, and is not hard-coded. For example, if the block editor is able to know that a "product" has a title and price attribute, we would also be able to know that.

So our UI renders the title and price "fields" (as it does today), and the user clicks on the title and on the price explicitly. The UI we have today, and the principle that the user should click on individual fields would remain.

So the html to blocks parser would be parsing individual fields, not the "container", which is the behavior we have today.

[akirk]

I agree it could be something similar to how block.json is layed out!

[ashfame]

I believe we would slowly converge into that idea, but starting right off the bat like that is hard as we encounter lots of unknowns in the implementation/architecture. It's like trying to write code in the first attempt that doesn't need refactoring, which is dreadfully slow. Much better to keep it moving and re-iterating over & over again :)

[adamziel]

Relevant: Explore Custom Post Types handling in Blueprint

[adamziel]

Actually, would it make any sense to use exactly the block.json file here? As in, not just a similar data format, but build an actual block.

[psrpinto]

would it make any sense to use exactly the block.json file here

I agree this is a possibility we should explore. I'm not sure we will be able to express all the things we need in a block.json, we can start with our own format, see where the similarities with block.json are, and then consider using a block.json.

[ashfame]

Thanks for writing the proposal! I like the idea of try-wp just being an alternative block editor but I can't imagine how it would even work in practice. The block editor concerns itself with the content of the page and the site editor concerns itself with the structure/layout of the page. Also note that not all pages in WordPress use block editor to manage content. Eg: Woocommerce's product page doesn't but might support in some other sense that I don't fully understand.

## Responsibility:
Block Editor <-> Content
Side Editor <-> Design (umbrella term)

The user interacts with the block editor for content of some pages (think a html page). Our objective however, is to liberate data by not just copying over the content of a html page from source site to new WordPress site, but carry over the relationships (media assets, hyperlinks etc) and auxiliary data (for backtracking & re-transforming the data) for each of that page. And even more importantly, orchestrate this process on scale for however many pages are there on the source site. This would involve working outside of the scope of a block editor, for crawling, establishing relationships and importing assets.

I think this complicates some practical aspects of achieving our objectives. Instead of working with the data as we understand it (direct access to under the hood), we would be limiting ourselves to only interface with how the Block editor operates, and when it can't support us in what we want to do, we would branch out and use custom endpoints anyway. Hence, I am unsure of the benefit offered by this approach.

Extensibility

I believe the process is best offered by sane defaults and then another candidate plugin can come in, at any point in future, to re-pick the data and re-transform data as many times as desired. This is aligned with we ourselves wanting to re-transform data at a later date if we have better transformations to offer.

We really shouldn't expect the user to make a choice of what e-commerce plugin they want to use for importing products, or even have them installed to begin with. This complicates the UX by introducing friction and offering more ways of how something can be done.

PHP-first approach

I believe a lot of solutions to tackle certain problems like (correcting links post import, downloading media assets and updating their reference links) is more suitable to be PHP-based to better address overlaps with other projects like Playground & to allow the possibility of eventually merging it in WP core.

I'm open to discussing this further and understanding your vision more deeply, as I may have missed some aspects of your proposal. Let me know your thoughts on these points.

[psrpinto]

Thanks for this feedback, you raise excellent points👌

---

The block editor concerns itself with the content of the page and the site editor concerns itself with the structure/layout of the page.

In the post I lumped block editor and site editor together because they behave in a similar way (both use blocks and talk to the WordPress API). However, as you mention, they deal with different types of data, so I agree we should look at those concerns (data vs layout) in different ways.

---

note that not all pages in WordPress use block editor to manage content. Eg: Woocommerce's product page doesn't

Indeed, one consequence of the approach I suggested would be that try-wordpress only supports blocks. Plugins that do not support blocks would not be able to integrate with try-wordpress. We could spin this in a positive light (it's a push for plugins to add support for the block editor), but in reality it's probably a deal-breaker. If try-wordpress doesn't work with one of the most-used WordPress plugins, that's no good.

---

I believe the process is best offered by sane defaults and then another candidate plugin can come in, at any point in future, to re-pick the data and re-transform data as many times as desired
We really shouldn't expect the user to make a choice of what e-commerce plugin they want to use

Agree with both these points. While that was what I was proposing in the post, I hadn't fully considered the implications. I wrote more about this in #92 (reply in thread).

---

This would involve working outside of the scope of a block editor, for crawling, establishing relationships and importing assets.

Things that deal with auxiliary data (i.e. data that is specific to our app and has no meaning outside of it) would indeed not be done in the scope of the block editor. This "block editor scope" only has to do with user data that comes from the source site.

Concerning importing assets, I would argue that this is the "block editor scope". If you would think about the equivalent action in the traditional block editor, that would be pasting some content that contains a link to an external image. So we would do the exact same thing: we submit the content to the WP API with a link to the image.

The downloading of the image would then be done in PHP, through a hook, or through a cron job, or through whatever other solution we deem adequate.

I believe a lot of solutions to tackle certain problems like (...) is more suitable to be PHP-based

The approach I was suggesting doesn't force us to do everything in the frontend. On the contrary, because we'd be using the Core REST API, we can fully leverage the WP ecosystem. For example, if there would be a plugin that hooks into saving a post and downloads external images that are referenced in the post's content, we could use that plugin.

[ashfame]

Alright cool!

Auxiliary data & work outside of block-editor scope

I think, this is where the most of the heavy-lifting would be done. I see the browser extension UI as a thin layer for grabbing context and most of the problems would be solved on the backend side for running transformations on raw data collected.

The approach I was suggesting doesn't force us to do everything in the frontend. On the contrary, because we'd be using the Core REST API, we can fully leverage the WP ecosystem.

One thing that stands out to me in this approach is that, now we would be trying to use all the REST APIs available to us, but we are only concerning ourselves with data at this point. So we can't use the endpoints we want to target directly since we want to save the raw/auxiliary data, so it must go through our custom endpoints resulting in the data populating at the right places. This seems contradictory.

[adamziel]

However, I think there is an alternative approach that would allow us to remove the need for this intermediate format, and bring along additional advantages. The remainder of this post will describe this alternative approach.

I love this. Any intermediate format is similar to reinventing WordPress exports, even if it's just for a brief moment in memory. The number 1 reason for using Playground here is not having to reinvent that wheel again. +100 for reusing the existing WordPress data structures and interfaces.

The original {Platform} to WordPress Proposal is highly relevant and the diagrams there overlap with the ones in this discussion.

[adamziel]

The UI should seamlessly adapt to the post type, so that it shows the fields that are relevant for that post type.

@dmsnell and I explored something related on CloudFest earlier this year:

• https://videopress.com/v/3gX5rYN7
• https://wordpress.com/blog/2024/04/15/custom-post-types-wordpress-admin/

There's also a related exploration from Automattic:

• https://github.com/Automattic/create-content-model

I believe something like this will ultimately become a part of WordPress core – it's such a popular problem.

[adamziel]

These explorations could help with using the block editor to manage structured data related to plugins that do not actually support blocks.

[psrpinto]

Thank you for these resources, will take a look!

[adamziel]

@youknowriad@mcsf@ellatrix Does anything in this discussion stand out to you as directionally aligned with core? Anything we can do to help each other?