Uh oh!
There was an error while loading. Please reload this page.
Feature deltas - #873
Conversation
…structure; begin documenting ChannelOptions changes, and setOptions
| h2(#using-params-with-lib-v11). Using channel params with v1.1 or earlier Ably client libraries | ||
| The current Ably libraries, at version 1.1, do not expose an API for expressing channel parameters. This means that it is necessary to specify parameters in a way that is opaque to the library. | ||
| The Ably libraries at version 1.1 and earlier, do not expose a the API for expressing channel parameters programmatically. This means that it is necessary to specify parameters in a way that is opaque to the library. |
There was a problem hiding this comment.
and earlier, do not expose
superflous comma
a the
| h4(#setting-channel). Setting channel options | ||
| A set of "channel options":#channel-options may also be passed to configure a channel for encryption. Find out more about "symmetric message encryption":/realtime/encryption. | ||
| A set of "channel options":#channel-options also be passed to specify options a channel when the channel is first obtained via "channels.get":#obtaining-channel. The options associated with a given channel may also be updated at any time after creation via "channel.setOptions":#setOptions |
There was a problem hiding this comment.
also be passed
to specify options a channel
There was a problem hiding this comment.
Also, the setOptions anchor doesn't appear on this page from what I can see.
| h2(#overview). Overview | ||
| Often a channel carries messages that represent a series updates to a particular object or document and, as such, there is a significant degree of similarily between successive messages. Delta mode is a way for a client to subscribe to a channel so that message payloads sent over the wire contain only the difference (ie the delta) between the present message and the previous message on the channel. The client can then apply the delta to the previous message to obtain the full payload. Using delta mode can signficantly reduce the encoded size of each message in the case that message payloads change by differences that are small relative to the size of the value. This reduction in size can reduce bandwidth costs, reduce transit latencies, and enable greater message throughput on a connection. |
| h2(#overview). Overview | ||
| Often a channel carries messages that represent a series updates to a particular object or document and, as such, there is a significant degree of similarily between successive messages. Delta mode is a way for a client to subscribe to a channel so that message payloads sent over the wire contain only the difference (ie the delta) between the present message and the previous message on the channel. The client can then apply the delta to the previous message to obtain the full payload. Using delta mode can signficantly reduce the encoded size of each message in the case that message payloads change by differences that are small relative to the size of the value. This reduction in size can reduce bandwidth costs, reduce transit latencies, and enable greater message throughput on a connection. |
There was a problem hiding this comment.
Maybe we could mention here that using the deltas would normally result in reducing the message size even if the subsequent messages are not mutations-of-an-object but random data. Deltas are compression in a way, i.e. any repeating blocks that the matching algorithm can find will be reduced to only a single repetition.
| Deltas are supported for realtime subscriptions only. Messages retrieved via the history API, and messages delivered to Reactor endpoints, are not compressed. Support for delta-compressed messages via Reactor is under consideration for the future. | ||
| Delta compression via @vcdiff@ is supported for all payloads, whether string or binary, or JSON-encoded. The delta algorithm processes message payloads as opaque binaries and has no dependency on the stucture of the payload - it does not process line-oriented diffs, for example. In principle, @vcdiff@ deltas can be applied to encrypted message payloads, but in practice this provides no benefit because there is no similarity between successive encrypted payloads even on identical or near-identical plaintext message payloads. |
There was a problem hiding this comment.
Again maybe the encrypted payloads statement is not true because of the delta compression thing.
There was a problem hiding this comment.
nah, properly encrypted data is incompressible. (In particular we use aes in cbc mode, not ecb; the encrypted payload will be indistinguishable from random data no matter how much repetition is in the plaintext)
| There is no constraint on how many publishers or subscribers there are. If there are multiple publishers, then deltas can still be generated, and they will be determined based on the order of messages in the channel in question, for the region in question. When there are multiple publishers in multiple regions, publishing messages nearly simultaneously, the ordering of messages delievred to subscribers can be different in different regions, depending on actual region-to-region transit latencies; in this case, deltas are generated based on the actual message order in each region, and subscribers are delivered a sequence of delta messages that reflects that regional order. | ||
| Delta processing, when activated on a channel, is performed for all messages on a channel, and deltas are calculated strictly based on the message ordering in that channel. Effectiveness of delta processing - that is, whether or not there is a material saving in payload size - is dependent on the level of similarity between successive payloads. Therefore, if a channel carries messages from multiple sources or streams that are dissimilar, then delta processing might not result in a useful size reduction, even if the messages in each individual stream are similar; it depends on the specific sequence of messages that occurs in the channel, in the region in question. |
There was a problem hiding this comment.
Again the size reduction might not be dependent on message similarity. However, we should probably mention here that the block matching we use for the deltas is not perfect because we are going for speed so size reduction is not guaranteed in all cases where it is possible.
| TBD | ||
| ``` | ||
| h3(#using-deltas-11). Via an Ably library before v1.2 (via a qualified channel name) |
There was a problem hiding this comment.
How are they going to apply the deltas in the pre 1.2 libs? In theory they could since the decoding will fail at vcdiff and the actual delta will get returned. Then they can use the OS libs we have to decode that. But that has to be documented maybe or there has to be reference to the non-ably usage section below.
| h2(#using-params-with-lib-v12). Using channel params with v1.2 or later Ably client libraries | ||
| Channel params may be specified in the @ChannelOptions@ when obtaining a @Channel@. A collection of channel params is expressed as a map of string key/value pairs. The @ChannelOptions@ associated with a channel may also be updated by calling "setOptions":./channel#setOptions. The params associated with a channel take effect when the channel is first attached; if the params are subsequently modified via a call to "setOptions":./channel#setOptions, then that call triggers attach operation that applies the updated params, if successful. |
There was a problem hiding this comment.
Clicking the 'setOptions' link I get:
File not found: /realtime/channel-params/channel
when viewing a local build (via bundle exec nanoc compile && bundle exec nanoc view -p 4000) at http://localhost:4000/realtime/channel-params/channel#setOptions
The Heroku preview presents that as:
404 Not Found : nginx
| h4(#setting-channel). Setting channel options | ||
| A set of "channel options":#channel-options may also be passed to configure a channel for encryption. Find out more about "symmetric message encryption":/realtime/encryption. | ||
| A set of "channel options":#channel-options also be passed to specify options a channel when the channel is first obtained via "channels.get":#obtaining-channel. The options associated with a given channel may also be updated at any time after creation via "channel.setOptions":#setOptions |
There was a problem hiding this comment.
Also, the setOptions anchor doesn't appear on this page from what I can see.
| h5(#setting-channel-params). Setting channel params | ||
| Channel parameters are a general mechanism by which a client can express properties of a channel, or of its attachment to a channel. The currently supported params are "rewind":./channel-params#rewind or "delta generation":./delta. |
There was a problem hiding this comment.
Neither of these links work. Both 404.
Out of interest why doesn't the build fail on this? Surely something in the docs build chain should be able to detect self-referential issues? cc @tomczoink@Ugbot
MarkWoulfeAbly
commented
May 12, 2020
This PR is being superseded by #880 |
This is the beginnings of the 1.2 docs updates relating to deltas and channel options/channel params. My aim is that this PR contains all of the information that will be needed by the docs team to complete this update.
In doing this I've speculatively broken up the old
channel-paramspage intochannel-params- explaining the mechanism itself - anddeltaandrewind. These need technical review, and then need some code snippets adding for the languages I haven't included.I also added some words for
Channel.setOptions, but someone needs to review if I've done it in the appropriate place, and there are also all the language-specific definitions needed for the method signature and params.@QuintinWillison I started this work from this branch, and I note that you subsequently merged Tsviatko's original
channel-paramscommit and made some changes. Those changes conflicted with my changes because I moved sections to new files, and the old deltas documentation is completely superseded. I have reviewed the changes that you made and I have tried to include those that are still relevant, but you should check that I did that faithfully. Sorry :(