Application: Machine Messaging on IPFS pubsub

[flyingzumwalt]

Work in progress - please contribute. See #40.

The basic concept: use IPFS + ipfs-based pubsub to manage things like queues that orchestrate machine workers instead of using centralized systems like redis or Amazon's queue + messaging services.

[daviddias]

@chrismatthieu has been paving the path with https://computes.io 🙌🏽

[chrismatthieu]

Thanks @diasdavid. @flyingzumwalt our blog has a bunch of cool videos of our IPFS-powered global neural net supercomputer. https://blog.computes.io

[hsanjuan]

A few months ago I forgot to add some notes back here:

Machine Messaging on IPFS pubsub

Machine messaging is a very wide topic ranges from traditional Message-Oriented middlewares, such a AMQP implementations or XMPP, to novel approaches to instant messaging and pubsub in distributed sytems. In very simple terms and making an wide definition, machine messaging [middleware] provides a layer which allows a set of producers to publish messages in the system. These messages are consumed or delivered to clients which are fully independent. They usually use queues (but not necessarily) to store the messages while they have not yet been consumed or delivered.

Usecases

• Data pipelining
• Orchestration
• Notifications
• Chat
• RPC
• Batch job scheduling

Which foundational features & functionality do the use cases need or rely on (ie. libp2p, private networks, etc.)?

• Asynchronicity: message can be sent and consumed separately. Producers and consumers are completely decoupled.
• Routing mechanisms: messages should be able to arrive to their destination/be received by consumers (libp2p-route?/dht routing?)
• Buffering: messages are queued until they are consumed/processed (ipfs-log?)
• Brokering: the system can perform more than just routing, for example, it can place messages in different queues, facilitating parallel consumption. pubsub's topics are a sort of brokerage but probably this should be more flexible.
• Some systems support request/response constructs (should be possible with libp2p)
• High-availability: messages can be pushed and pulled from different nodes. Nodes downtimes don't interrupt the system's capacity to deliver messages (libp2p + routing should help here).
• Web-scale ¯_(ツ)_/¯

Related products or projects that are addressing those use cases

• The dozens of AMQP implementations: ZeroMQ, ApacheMQ
• Apache Kafka, as a distributed streaming platform. Used in every "big data" place.
• Linux desktops and their d-bus implementations
• Amazon SQS
• The dozens of XMPP implementations
• Matrix protocol for secure instant messaging
• Ethereum's Whisper (used in Akasha)

Related organizations who either 1) have those use cases or 2) are building solutions for those use cases

• Anyone working primarily with data and processing (big data and small data) is probably using a messaging system of some sort.
• Anyone doing chats needs a messaging solution. Usually centralized, but we've also seen Matrix, Whisper, Orbit
• Akasha has already given use to Ethereum's pubsub system Whisper
• It's not hard to imagine OpenBazaar and others would benefit for some sort of chat, but also things like auctioning could rely on pubsub systems

[daviddias]

I believe that @mcollina would be interested in this convo! :)

Hi Matteo! Are you still doing stuff with MQTT and Mosca? Have you heard about IPFS? Familiar with the work that @chrismatthieu is doing with Computes.io?

I'm interested in learning how pluggable MQTT over libp2p would be? At first glance it seems it is very dependable of the low level transport, but if we could mount MQTT control and message streams over any duplex stream, then we can use libp2p to establish the P2P connection and MQTT to do the brokering system. Thanks in advance :)

[mcollina]

Hey @diasdavid, MQTT needs a binary duplex channel to run. Can this be achieved on top of libp2p?

[daviddias]

Absolutely it can! IPFS uses it that way to transfer files as blocks of data and other apps have been using it in the same way for other purposes.

We can mount any protocol in libp2p using Protocol Multiplexing, saving the hurdle of open extra connections between peers which are quite expensive in a P2P network. @pgte has also been doing a lot of work of using libp2p pubsub to distribute state events for CRDTs and one of the demos I did in the past was loading the Ethereum Blockchain on the browser using libp2p.

[daviddias]

Folks at IDEO have been using libp2p too. @ReidWilliamsm, @gavinmcdermott you might be interested in following this thread :)

[mcollina]

Protocol multiplexing seems message oriented. Can it be mapped 1-1 with a Unix socket? Or code-wise, can it be implemented as a Duplex in Node? If that's the case, you can already use MQTT on top via https://github.com/mqttjs/MQTT.js#client.

[chrismatthieu]

We still use @mcollina Mosca MQTT broker on Octoblu and love it. I would gladly incorporate his MQTT/IPFS implementation into Computes!

[chrismatthieu]

@mcollina IPFS supports pubsub. Nodes can listen and broadcast to channels and even setup P2P channels by their IPFS node IDs. It doesn't have the concept of QoS. This would be very important to Computes.

There's a very interesting IPFS database project underway called OrbitDB (https://github.com/orbitdb/orbit-db). You may be able to use this DB for tracking delivery like an MQTT broker.

[agahEbrahimi]

Are there any Golang examples of this?
Thank you

[ozra]

Following with interest :)

[oDinZu]

Hush blockchain integration may be an idea. https://github.com/MyHush

Hushlist: Censorship Resistant Metadata-Minimizing Multi-Blockchain Communication
https://github.com/leto/hushlist