Twitter + RSS + Markdown

[scripting]

First check out the new features to support Markdown in RSS, along with hooking Twitter up to RSS.

If you have any questions or comments, please post them here.

Remember to be professional and respectful. 😄

[jeffevans108]

not a question — since the get go, RSS is the Bling! tnx a gazillion Dave

[scripting]

@jeffevans108 -- keep on truckin! ;-)

[rsdoiel]

This is really inspiring, thank you.

I need some advise on approach. I assemble by blog from Markdown documents. Some have frontmatter (i.e. a YAML block at the top of the document with metadata about the article). In your source:markdown example would you include the whole document or omit frontmatter?

[scripting]

@rsdoiel -- i'm glad you find this inspiring. I do too, esp the opportunity to work with smart people on something that matters. ;-)

I can't answer this question without more information.

Does your blog have an RSS feed? If so it would be helpful to see it and the source text with the YAML block.

[rsdoiel]

Here's the gory details that got me thinking about front matter and Markdown. You can see an example blog post
at

---
title: "Installing Golang from source on RPi-OS for arm64"
series: "Raspberry Pi"
number: 1
author: "R. S. Doiel"
date: "2022-02-18"
copyright: "copyright (c) 2022, R. S. Doiel"
keywords: [ "raspberry pi", "Raspberry Pi OS", "arm64" ]
license: "https://creativecommons.org/licenses/by-sa/4.0/"
---
Installing Golang from Source on RPi-OS for arm64
==========================================
By R. S. Doiel, 2022-02-18
This are my quick notes on installing Golang from source on the Raspberry Pi OS 64 bit.
1. Get a working compiler
a. go to https://go.dev/dl/ and download go1.17.7.linux-arm64.tar.gz
b. untar the tarball in your home directory (it'll unpack to $HOME/go)
c. `cd go/src` and `make.bash`
2. Move go directory to go1.17
3. Clone go from GitHub
4. Compile with the downloaded compiler
a. `cd go/src`
b. `env GOROOT_BOOTSTRAP=$HOME/go1.17 ./make.bash`
c. Make sure `$HOME/go/bin` is in the path
d. `go version`

The frontmatter is delimited by "---" (opening) and "---" (closing) alone on the line. This is common practice for people using Pandoc (me) and RMarkdown (colleagues at a small science institute where I work).

The section of the RSS I think would be updated is below (I truncated the copy I made from the RSS link above). Warning this looks really ugly, something I wasn't happy with but didn't have a better idea until I read your post on the source name spaces and what you've been up to with reallysimple.

<?xml version="1.0"?>
<rss version="2.0">
<channel>
<title>R. S. Doiel</title>
<link>https://rsdoiel.github.io</link>
<description>Robert&amp;#39;s ramblings and wonderigs</description>
<pubDate>Tue, 19 Jul 2022 00:00:00 UTC</pubDate>
<lastBuildDate>Tue, 19 Jul 2022 00:00:00 UTC</lastBuildDate>
<generator>mkrss 1.0.2</generator>
<item>
<title>Installing Golang from source on RPi-OS for arm64</title>
<link>/blog/2022/02/18/Installing-Go-from-Source-RPiOS-arm64.md</link>
<description>&#xA;&#xA;Installing Golang from Source on RPi-OS for arm64&#xA;==========================================&#xA;&#xA;By R. S. Doiel, 2022-02-18&#xA;&#xA;This are my quick notes on installing Golang from source on the Raspberry Pi OS 64 bit.&#xA;&#xA;1. Get a working compiler&#xA;&#x9;a. go to https://go.dev/dl/ and download go1.17.7.linux-arm64.tar.gz&#xA;&#x9;b. untar the tarball in your home directory (it&#39;ll unpack to $HOME/go)&#xA;&#x9;c. `cd go/src` and `make.bash`&#xA;2. Move go directory to go1.17&#xA;3. Clone go from GitHub&#xA;4. Compile with the downloaded compiler&#xA;&#x9;a. `cd go/src`&#xA;&#x9;b. `env GOROOT_BOOTSTRAP=$HOME/go1.17 ./make.bash`&#xA;&#x9;c. Make sure `$HOME/go/bin` is in the path&#xA;&#x9;d. `go version` ...</description>
<pubDate>Fri, 18 Feb 2022 00:00:00 UTC</pubDate>
<guid>/blog/2022/02/18/Installing-Go-from-Source-RPiOS-arm64.md</guid>
</item>
<!-- ... and the rest of the feed follows -->

The YAML front matter seems like it should be snipped before if the primary source:markdown target is display but if the source:markdown is intended as an ingest field then it should include the whole document (because there is useful metadata about the post). That is where I started to scratch my head and realize I wasn't quite getting it.

Source urls --

Markdown version:
https://raw.githubusercontent.com/rsdoiel/rsdoiel.github.io/main/blog/2022/02/18/Installing-Go-from-Source-RPiOS-arm64.md

HTML verison:
https://rsdoiel.github.io/blog/2022/02/18/Installing-Go-from-Source-RPiOS-arm64.html

RSS:
https://rsdoiel.github.io/rss.xml

All the best,

Robert

[scripting]

@rsdoiel -- thanks for providing the detail.

The frontmatter is playing the role of RSS, so it should not be part of the RSS item. Just the text of the post itself, starting with "Installing Golang from Source..."

Some of the stuff in the frontmatter belongs at the top of your RSS feed, like the copyright.

You are repeating the title of the post in the RSS item and in the Markdown text. Most if not all RSS readers will repeat the title if you do it that way. Not a very good user experience.

I guess the summary is that the YAML stuff is doing what RSS is doing...

THe best way to proceed imho is to try feeding your RSS into a typical feed reader and see what you get, and if you can, go back and revise the code that generates the RSS iteratively until you get something reasonable showing up in the reader.

[rsdoiel]

Thank you so much for the pointers! I'm rewriting the tools I use and this is a big help and inspiration.

[scripting]

BTW -- I just posted devnotes about Markdown in feeds.

http://scripting.com/2022/07/19/152235.html?title=devNotesForMarkdownInFeeds

[colin-walker]

I think I've got it. My ignorance getting the better of me.

[scripting]

OK if you have questions, post them in the thread i pointed to above.

[scripting]

And if you get it working, please post an url so i can subscribe.

[colin-walker]

Seemed to work okay, the log showed the ping and it recognised an update 🤞

The feed is here.

I'm going to be blogging properly again from the 25th so you'll have to let me know if it is all okay.

[scripting]

If you post this in the thread I started more people who can help will see it. I’m not the expert on this.

[josephzitt]

One question: It's very cool that you're supporting Markdown -- but which Markdown are you using? There appear to be a lot of variants, many of which think that they are the One True Markdown. When you discuss it in the blog, could you link to a specification or something, to avoid confusion (such as mine)?

[scripting]

@josephzitt -- of course as you know there is no One True Markdown.

I think the approach we all take with Markdown text is to stay within a reasonable subset of the syntax, and hope for the best. ;-)

I go by this cheat sheet, which I keep in an icon in the chrome of my browser, easy to reach for. And it seems to work.

Re the Markdown processors I use --

1. On the client side -- I use something called MarkdownCoverter, which I downloaded many years ago. I seem to recall it's the one that StackExchange uses, so I figured that's got to be pretty good.

2. On the server side -- I use markd. I don't recall how I decided to use this one, but it has served me well.

As I understand it Gruber like the way it has evolved. The shit does kind of hit the fan when it comes to using Markdown in something like RSS where interop is the primary driving force. But this was one of the first enhancements I wanted to get out there in 2022, the 20th year of RSS 2.0. 😄

PS: I made this into a post on my blog.

[pzingg]

@scripting FYI I just submitted a pull request in the Akkoma (Mastodon-compatible server) repository to add <source:markdown> to their RSS feeds: https://akkoma.dev/AkkomaGang/akkoma/pulls/343

If I have time, I may try to do the same for glitch-soc. I saw you have a GitHub issue there.

I also have a question on how Mastodon, as a feed producer, should set the <cloud> element. I see that the way WordPress.com implements it, the rssCloud server is set to the same domain as the originating WordPress blog (they must have implemented their own rssCloud server). In the Mastodon server case, do you think that the rssCloud server domain should be a single setting for the entire server (some of these servers have 500K users), or a per-user (per-feed) preference?

I can probably hack Akkoma into supporting rssCloud, by implementing the functionality of Andrew Shell's example in Elixir. Then there could be a server-wide setting to use either Akkoma's own implementation, or any other public server (like rpc.rsscloud.org if it's still active).

[scripting]

@pzingg --

thanks for putting out the word on souce:markdown support. once we get a little critical mass there it should get really interesting. ;-)

i would much prefer it not be a feed-level preference setting, and that mastodon servers include their own cloud server support.

or we could see what @andrewshell thinks -- i don't run a cloud server myself, i just use his.

[andrewshell]

Yeah, I’m guessing if a substantial number of servers start pinging my instance it will probably break. Best case is if is handled by mastodon. It’s a super simple protocol and what mastodon is doing already is probably pretty similar.

[pzingg]

Sadly I doubt the Akkoma gang will accept my PR.

In submitting the PR I have some questions for you about recommendations for escaping characters like quotes, ampersands, angle brackets, newlines and “&”-like entities in RSS. Would the escaping be the same for HTML and Markdown? I found a bunch of articles from 2008 to the present that seem to indicate that feed readers don’t unescape content (title, description) the same way. Are there best practices?

Also I am making progress in hooking up RSSCloud “ping” messages to Akkoma status updates in real time. Did you know that Mastodon API has a websocket stream for updates?

https://docs.joinmastodon.org/methods/streaming/#web-socket

I was able to listen to the public firehose of updates for an Akkoma server with the “public:local” stream. Now to filter those updates by subscribed user feeds and then send a ping to the little Elixir RSSCloud server I’m building. All this can be done outside of the Mastodon/Akkoma server.

[scripting]

@pzingg -- I recommend using my daverss package as prior art.

Do what it does. I never get complaints about it not working with feed readers.

https://github.com/scripting/rss

[pzingg]

I just today added three @scripting inspired issues to Akkoma (a Mastodon API-compatible server), and I can try to do the same for either Mastodon or one of its forks that supports Markdown, namely glitch-soc. The main Mastodon repository still only support plain text in posts, whereas glitch-soc also supports posts authored in HTML and Markdown.

If you have any more firepower to add to the "Idea" or "Reasoning" sections of these issues, please expand here, and I'll amend the issues. I have pull requests ready to go for all these items...

• Support RSS Cloud
• Support title-less feed items
• Support round-trip Markdown (and other content types)