Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
246 changes: 246 additions & 0 deletions docs/RFCs/personal-data-reserve.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
# Personal Data Reserve
- Start Date: 2022-03-18
- RFC PR: [functionland/docs/pull/61](https://github.com/functionland/docs/pull/61)
- Functionland Issue: [functionland/docs/issues/58](https://github.com/functionland/docs/issues/58)
- Status: Draft
- Authors: [Aaron Surty](https://github.com/gitaaron),
- Reviewers: [Farhoud](https://github.com/farhoud), [Ehsan](https://github.com/ehsan6sha), [Masih](https://github.com/orgs/functionland/people/masih)

## Summary
[summary]: #summary

This RFC covers how a BOX customer's data can be replicated across a group of BOXes physically separated from each other to help prevent unwanted loss.

## Use Case
[use-case]: #use-case

A person owns 3-5 BOXes (eg/ one in their home, one at their office and one at a friend's house).

Their basement floods bricking the one located in their home.

They are still able to access all of the data that was uploaded to their bricked home BOX from one of their other BOXes or a brand new BOX provisioned with factory settings.

The following scenarios are handled:

* adding a new BOX to the reserve

* removing a BOX from the reserve

* data becomes corrupted on a BOX


## Terminology
[terminology]: #terminology

| Name | Definition |
|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
| reserve | a group of keepers collaboratively working together to back up a snapshot |
| snapshot | a collection of data that represents the entire file system being backed up at a given moment in time |
| chunk | a portion of a data set |
| keeper | a BOX process responsible for backing up a file system to the reserve |
| retriever | a BOX process given the task of rebuilding a usable file system from a snapshot |
| author | the BOX where the backed up data set was created |
| peer | a member of the reserve |
| replication factor | how many keepers a chunk of data is stored on; a greater replication factor means greater resilience |
| data set | a file, directory, or data from a database |
| warning time window | how much time should be given to send an alert before an imminent limitation is encountered and a system failure occurs |



## Pre-conditions
[pre-conditions]: #pre-conditions

* the owner has already authenticated themselves with each BOX

* the fula-api is persisting data to a disk accessible by the keeper

* each BOX is already provisioned with the necessary configuration info required to operate a reserve

## Assumptions

* each keeper in a reserve can be trusted to not operate maliciously by performing unwanted delete operations

* the data being backed up is:
* multimedia files such as photos and video
* [orbit db](https://orbitdb.org/) metadata
* plain text files for shared configuration data

## Out of Scope

* high-frequency updates to data sets which might be encountered with multi-tenancy or recording streaming video

* syncing of data between BOXes so that it is usable on each BOX (eg/ CRUD operations on them in real time)

* conflicts arising from concurrent updates initiated by a user on several different BOXes

* heuristics used to minimize bandwidth and improve availability

## Limitations
[limitations]: #limitations

The following limitations may be encountered while operating a reserve:
* file size
* number of files in a directory
* snapshot size
* number of keepers in a reserve

## Configuration
[configuration]: #configuration

Configuration data for each peer can be split into local and shared.

### Local Configuration

* local BOX address

* local public/private key

### Shared Configuration

* remote BOX addresses of other peers in the reserve

* shared secret

* minimum acceptable replication factor

* 'normal' event dispatching frequency

* warning time window
* has a global default as well as an override for each limitation

## Conflict Resolution
[conflict-resolution]: #conflict-resolution

Although it can be assumed the snapshot is already free from conflicts caused by concurrent updates initiated from a user, conflicts may still arise from disk corruption or other unforseen keeper errors.

The disputing keepers will take the appropriate steps to resolve the conflict. If an appropriate resolution can not be achieved, an event is raised.

## Events
The following events are dispatched for an administrative UI.

* limitation imminent

* limitation encountered

* keeper health
* memory
* disk I/O
* CPU usage
* disk corruption

* unresolved conflict

* unacceptable replication factor

* keeper added / removed

* network disruption

### Event Types
* normal
* warning
* failure

### Warnings

Events dispatched before a failure occurs based on a forecasting heuristic to determine how quickly a limit will be reached.

### Event Frequency

Normal events are dispatched periodically (based on a config param) for historical reporting and warnings|failures are dispatched immediately.

## Implementation
[reference-level-explanation]: #reference-level-explanation

### Components

The components that are needed for this use case are:

* data set keeper

* data set retriever

* event dispatcher

* file integrity monitor

### Event Dispatcher

A mechanism for dispatching events and queueing them to prevent loss if a peer goes down.

### File Integrity Monitoring

For detecting / handling disk corruption.

A full sweep of the file system is periodically done comparing the contents of chunks on disk with a source of truth.


### Network Architecture

A peer-peer architecture is used over master/slave so that if a single BOX goes down the rest of the reserve will still be able to operate normally.

* any shared config data is stored on each BOX

* any shared state required for the retrieval of a data set is stored on each BOX

* no central servers are used for networking


## Drawbacks
[drawbacks]: #drawbacks
Putting the responsibility of data reliability on BOX owners means there is potential for a BOX owner to make a mistake and permanently lose their data.

## Rationale and alternatives
[rationale-and-alternatives]: #rationale-and-alternatives
An alternative could be to use paid services (cloud storage providers) with their own SLAs to take on the responsibility of data reliability.

If participating in a decentralized storage network (DSN), the BOX owner could also purchase a mining component to offset their cost.

There are currently a few drawbacks with this:

* becoming a storage miner requires a significant upfront investment to cover hardware and staking costs

* a private reserve will always be more efficient since keepers will not have to worry about the overhead of trusting each other

These options are not mutually exclusive. Offering both options (free and paid) could provide the greatest freedom/flexibility for BOX owners.

## Dependencies

* [IPFS cluster](https://cluster.ipfs.io/)

* private connections

* authentication

* retriever

* keeper

@TODO - provide links to relevant docs

## Unresolved questions
[unresolved-questions]: #unresolved-questions

What is an ideal replication factor?

How can resilience be measured? Markov models?

How can contents of an entire filesystem be efficiently compared with a source of truth?

How can system resources and their limits be calculated?

How can NAT hole punching work in a pnet without any relays?

Should data compression be taken into account?

## Future Possibilities
[future-possibilities]: #future-possibilities

### History

Storing a history snapshots so an owner can go back in time and recover a data set from a previous state. This could help with data loss due to user errors.

### Regions

A person or group of people may have a large number of BOXes and want to set up geographically grouped regions such that if there is a severe network outage, each region will still be able to recover the entire contents of a snapshot.

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
246 changes: 246 additions & 0 deletions docs/RFCs/personal-data-reserve.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
# Personal Data Reserve
- Start Date: 2022-03-18
- RFC PR: [functionland/docs/pull/61](https://github.com/functionland/docs/pull/61)
- Functionland Issue: [functionland/docs/issues/58](https://github.com/functionland/docs/issues/58)
- Status: Draft
- Authors: [Aaron Surty](https://github.com/gitaaron),
- Reviewers: [Farhoud](https://github.com/farhoud), [Ehsan](https://github.com/ehsan6sha), [Masih](https://github.com/orgs/functionland/people/masih)

## Summary
[summary]: #summary

This RFC covers how a BOX customer's data can be replicated across a group of BOXes physically separated from each other to help prevent unwanted loss.

## Use Case
[use-case]: #use-case

A person owns 3-5 BOXes (eg/ one in their home, one at their office and one at a friend's house).

Their basement floods bricking the one located in their home.

They are still able to access all of the data that was uploaded to their bricked home BOX from one of their other BOXes or a brand new BOX provisioned with factory settings.

The following scenarios are handled:

* adding a new BOX to the reserve

* removing a BOX from the reserve

* data becomes corrupted on a BOX


## Terminology
[terminology]: #terminology

| Name | Definition |
|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
| reserve | a group of keepers collaboratively working together to back up a snapshot |
| snapshot | a collection of data that represents the entire file system being backed up at a given moment in time |
| chunk | a portion of a data set |
| keeper | a BOX process responsible for backing up a file system to the reserve |
| retriever | a BOX process given the task of rebuilding a usable file system from a snapshot |
| author | the BOX where the backed up data set was created |
| peer | a member of the reserve |
| replication factor | how many keepers a chunk of data is stored on; a greater replication factor means greater resilience |
| data set | a file, directory, or data from a database |
| warning time window | how much time should be given to send an alert before an imminent limitation is encountered and a system failure occurs |



## Pre-conditions
[pre-conditions]: #pre-conditions

* the owner has already authenticated themselves with each BOX

* the fula-api is persisting data to a disk accessible by the keeper

* each BOX is already provisioned with the necessary configuration info required to operate a reserve

## Assumptions

* each keeper in a reserve can be trusted to not operate maliciously by performing unwanted delete operations

* the data being backed up is:
* multimedia files such as photos and video
* [orbit db](https://orbitdb.org/) metadata
* plain text files for shared configuration data

## Out of Scope

* high-frequency updates to data sets which might be encountered with multi-tenancy or recording streaming video

* syncing of data between BOXes so that it is usable on each BOX (eg/ CRUD operations on them in real time)

* conflicts arising from concurrent updates initiated by a user on several different BOXes

* heuristics used to minimize bandwidth and improve availability

## Limitations
[limitations]: #limitations

The following limitations may be encountered while operating a reserve:
* file size
* number of files in a directory
* snapshot size
* number of keepers in a reserve

## Configuration
[configuration]: #configuration

Configuration data for each peer can be split into local and shared.

### Local Configuration

* local BOX address

* local public/private key

### Shared Configuration

* remote BOX addresses of other peers in the reserve

* shared secret

* minimum acceptable replication factor

* 'normal' event dispatching frequency

* warning time window
* has a global default as well as an override for each limitation

## Conflict Resolution
[conflict-resolution]: #conflict-resolution

Although it can be assumed the snapshot is already free from conflicts caused by concurrent updates initiated from a user, conflicts may still arise from disk corruption or other unforseen keeper errors.

The disputing keepers will take the appropriate steps to resolve the conflict. If an appropriate resolution can not be achieved, an event is raised.

## Events
The following events are dispatched for an administrative UI.

* limitation imminent

* limitation encountered

* keeper health
* memory
* disk I/O
* CPU usage
* disk corruption

* unresolved conflict

* unacceptable replication factor

* keeper added / removed

* network disruption

### Event Types
* normal
* warning
* failure

### Warnings

Events dispatched before a failure occurs based on a forecasting heuristic to determine how quickly a limit will be reached.

### Event Frequency

Normal events are dispatched periodically (based on a config param) for historical reporting and warnings|failures are dispatched immediately.

## Implementation
[reference-level-explanation]: #reference-level-explanation

### Components

The components that are needed for this use case are:

* data set keeper

* data set retriever

* event dispatcher

* file integrity monitor

### Event Dispatcher

A mechanism for dispatching events and queueing them to prevent loss if a peer goes down.

### File Integrity Monitoring

For detecting / handling disk corruption.

A full sweep of the file system is periodically done comparing the contents of chunks on disk with a source of truth.


### Network Architecture

A peer-peer architecture is used over master/slave so that if a single BOX goes down the rest of the reserve will still be able to operate normally.

* any shared config data is stored on each BOX

* any shared state required for the retrieval of a data set is stored on each BOX

* no central servers are used for networking


## Drawbacks
[drawbacks]: #drawbacks
Putting the responsibility of data reliability on BOX owners means there is potential for a BOX owner to make a mistake and permanently lose their data.

## Rationale and alternatives
[rationale-and-alternatives]: #rationale-and-alternatives
An alternative could be to use paid services (cloud storage providers) with their own SLAs to take on the responsibility of data reliability.

If participating in a decentralized storage network (DSN), the BOX owner could also purchase a mining component to offset their cost.

There are currently a few drawbacks with this:

* becoming a storage miner requires a significant upfront investment to cover hardware and staking costs

* a private reserve will always be more efficient since keepers will not have to worry about the overhead of trusting each other

These options are not mutually exclusive. Offering both options (free and paid) could provide the greatest freedom/flexibility for BOX owners.

## Dependencies

* [IPFS cluster](https://cluster.ipfs.io/)

* private connections

* authentication

* retriever

* keeper

@TODO - provide links to relevant docs

## Unresolved questions
[unresolved-questions]: #unresolved-questions

What is an ideal replication factor?

How can resilience be measured? Markov models?

How can contents of an entire filesystem be efficiently compared with a source of truth?

How can system resources and their limits be calculated?

How can NAT hole punching work in a pnet without any relays?

Should data compression be taken into account?

## Future Possibilities
[future-possibilities]: #future-possibilities

### History

Storing a history snapshots so an owner can go back in time and recover a data set from a previous state. This could help with data loss due to user errors.

### Regions

A person or group of people may have a large number of BOXes and want to set up geographically grouped regions such that if there is a severe network outage, each region will still be able to recover the entire contents of a snapshot.

, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
246 changes: 246 additions & 0 deletions docs/RFCs/personal-data-reserve.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
# Personal Data Reserve
- Start Date: 2022-03-18
- RFC PR: [functionland/docs/pull/61](https://github.com/functionland/docs/pull/61)
- Functionland Issue: [functionland/docs/issues/58](https://github.com/functionland/docs/issues/58)
- Status: Draft
- Authors: [Aaron Surty](https://github.com/gitaaron),
- Reviewers: [Farhoud](https://github.com/farhoud), [Ehsan](https://github.com/ehsan6sha), [Masih](https://github.com/orgs/functionland/people/masih)

## Summary
[summary]: #summary

This RFC covers how a BOX customer's data can be replicated across a group of BOXes physically separated from each other to help prevent unwanted loss.

## Use Case
[use-case]: #use-case

A person owns 3-5 BOXes (eg/ one in their home, one at their office and one at a friend's house).

Their basement floods bricking the one located in their home.

They are still able to access all of the data that was uploaded to their bricked home BOX from one of their other BOXes or a brand new BOX provisioned with factory settings.

The following scenarios are handled:

* adding a new BOX to the reserve

* removing a BOX from the reserve

* data becomes corrupted on a BOX


## Terminology
[terminology]: #terminology

| Name | Definition |
|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
| reserve | a group of keepers collaboratively working together to back up a snapshot |
| snapshot | a collection of data that represents the entire file system being backed up at a given moment in time |
| chunk | a portion of a data set |
| keeper | a BOX process responsible for backing up a file system to the reserve |
| retriever | a BOX process given the task of rebuilding a usable file system from a snapshot |
| author | the BOX where the backed up data set was created |
| peer | a member of the reserve |
| replication factor | how many keepers a chunk of data is stored on; a greater replication factor means greater resilience |
| data set | a file, directory, or data from a database |
| warning time window | how much time should be given to send an alert before an imminent limitation is encountered and a system failure occurs |



## Pre-conditions
[pre-conditions]: #pre-conditions

* the owner has already authenticated themselves with each BOX

* the fula-api is persisting data to a disk accessible by the keeper

* each BOX is already provisioned with the necessary configuration info required to operate a reserve

## Assumptions

* each keeper in a reserve can be trusted to not operate maliciously by performing unwanted delete operations

* the data being backed up is:
* multimedia files such as photos and video
* [orbit db](https://orbitdb.org/) metadata
* plain text files for shared configuration data

## Out of Scope

* high-frequency updates to data sets which might be encountered with multi-tenancy or recording streaming video

* syncing of data between BOXes so that it is usable on each BOX (eg/ CRUD operations on them in real time)

* conflicts arising from concurrent updates initiated by a user on several different BOXes

* heuristics used to minimize bandwidth and improve availability

## Limitations
[limitations]: #limitations

The following limitations may be encountered while operating a reserve:
* file size
* number of files in a directory
* snapshot size
* number of keepers in a reserve

## Configuration
[configuration]: #configuration

Configuration data for each peer can be split into local and shared.

### Local Configuration

* local BOX address

* local public/private key

### Shared Configuration

* remote BOX addresses of other peers in the reserve

* shared secret

* minimum acceptable replication factor

* 'normal' event dispatching frequency

* warning time window
* has a global default as well as an override for each limitation

## Conflict Resolution
[conflict-resolution]: #conflict-resolution

Although it can be assumed the snapshot is already free from conflicts caused by concurrent updates initiated from a user, conflicts may still arise from disk corruption or other unforseen keeper errors.

The disputing keepers will take the appropriate steps to resolve the conflict. If an appropriate resolution can not be achieved, an event is raised.

## Events
The following events are dispatched for an administrative UI.

* limitation imminent

* limitation encountered

* keeper health
* memory
* disk I/O
* CPU usage
* disk corruption

* unresolved conflict

* unacceptable replication factor

* keeper added / removed

* network disruption

### Event Types
* normal
* warning
* failure

### Warnings

Events dispatched before a failure occurs based on a forecasting heuristic to determine how quickly a limit will be reached.

### Event Frequency

Normal events are dispatched periodically (based on a config param) for historical reporting and warnings|failures are dispatched immediately.

## Implementation
[reference-level-explanation]: #reference-level-explanation

### Components

The components that are needed for this use case are:

* data set keeper

* data set retriever

* event dispatcher

* file integrity monitor

### Event Dispatcher

A mechanism for dispatching events and queueing them to prevent loss if a peer goes down.

### File Integrity Monitoring

For detecting / handling disk corruption.

A full sweep of the file system is periodically done comparing the contents of chunks on disk with a source of truth.


### Network Architecture

A peer-peer architecture is used over master/slave so that if a single BOX goes down the rest of the reserve will still be able to operate normally.

* any shared config data is stored on each BOX

* any shared state required for the retrieval of a data set is stored on each BOX

* no central servers are used for networking


## Drawbacks
[drawbacks]: #drawbacks
Putting the responsibility of data reliability on BOX owners means there is potential for a BOX owner to make a mistake and permanently lose their data.

## Rationale and alternatives
[rationale-and-alternatives]: #rationale-and-alternatives
An alternative could be to use paid services (cloud storage providers) with their own SLAs to take on the responsibility of data reliability.

If participating in a decentralized storage network (DSN), the BOX owner could also purchase a mining component to offset their cost.

There are currently a few drawbacks with this:

* becoming a storage miner requires a significant upfront investment to cover hardware and staking costs

* a private reserve will always be more efficient since keepers will not have to worry about the overhead of trusting each other

These options are not mutually exclusive. Offering both options (free and paid) could provide the greatest freedom/flexibility for BOX owners.

## Dependencies

* [IPFS cluster](https://cluster.ipfs.io/)

* private connections

* authentication

* retriever

* keeper

@TODO - provide links to relevant docs

## Unresolved questions
[unresolved-questions]: #unresolved-questions

What is an ideal replication factor?

How can resilience be measured? Markov models?

How can contents of an entire filesystem be efficiently compared with a source of truth?

How can system resources and their limits be calculated?

How can NAT hole punching work in a pnet without any relays?

Should data compression be taken into account?

## Future Possibilities
[future-possibilities]: #future-possibilities

### History

Storing a history snapshots so an owner can go back in time and recover a data set from a previous state. This could help with data loss due to user errors.

### Regions

A person or group of people may have a large number of BOXes and want to set up geographically grouped regions such that if there is a severe network outage, each region will still be able to recover the entire contents of a snapshot.

, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
246 changes: 246 additions & 0 deletions docs/RFCs/personal-data-reserve.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
# Personal Data Reserve
- Start Date: 2022-03-18
- RFC PR: [functionland/docs/pull/61](https://github.com/functionland/docs/pull/61)
- Functionland Issue: [functionland/docs/issues/58](https://github.com/functionland/docs/issues/58)
- Status: Draft
- Authors: [Aaron Surty](https://github.com/gitaaron),
- Reviewers: [Farhoud](https://github.com/farhoud), [Ehsan](https://github.com/ehsan6sha), [Masih](https://github.com/orgs/functionland/people/masih)

## Summary
[summary]: #summary

This RFC covers how a BOX customer's data can be replicated across a group of BOXes physically separated from each other to help prevent unwanted loss.

## Use Case
[use-case]: #use-case

A person owns 3-5 BOXes (eg/ one in their home, one at their office and one at a friend's house).

Their basement floods bricking the one located in their home.

They are still able to access all of the data that was uploaded to their bricked home BOX from one of their other BOXes or a brand new BOX provisioned with factory settings.

The following scenarios are handled:

* adding a new BOX to the reserve

* removing a BOX from the reserve

* data becomes corrupted on a BOX


## Terminology
[terminology]: #terminology

| Name | Definition |
|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
| reserve | a group of keepers collaboratively working together to back up a snapshot |
| snapshot | a collection of data that represents the entire file system being backed up at a given moment in time |
| chunk | a portion of a data set |
| keeper | a BOX process responsible for backing up a file system to the reserve |
| retriever | a BOX process given the task of rebuilding a usable file system from a snapshot |
| author | the BOX where the backed up data set was created |
| peer | a member of the reserve |
| replication factor | how many keepers a chunk of data is stored on; a greater replication factor means greater resilience |
| data set | a file, directory, or data from a database |
| warning time window | how much time should be given to send an alert before an imminent limitation is encountered and a system failure occurs |



## Pre-conditions
[pre-conditions]: #pre-conditions

* the owner has already authenticated themselves with each BOX

* the fula-api is persisting data to a disk accessible by the keeper

* each BOX is already provisioned with the necessary configuration info required to operate a reserve

## Assumptions

* each keeper in a reserve can be trusted to not operate maliciously by performing unwanted delete operations

* the data being backed up is:
* multimedia files such as photos and video
* [orbit db](https://orbitdb.org/) metadata
* plain text files for shared configuration data

## Out of Scope

* high-frequency updates to data sets which might be encountered with multi-tenancy or recording streaming video

* syncing of data between BOXes so that it is usable on each BOX (eg/ CRUD operations on them in real time)

* conflicts arising from concurrent updates initiated by a user on several different BOXes

* heuristics used to minimize bandwidth and improve availability

## Limitations
[limitations]: #limitations

The following limitations may be encountered while operating a reserve:
* file size
* number of files in a directory
* snapshot size
* number of keepers in a reserve

## Configuration
[configuration]: #configuration

Configuration data for each peer can be split into local and shared.

### Local Configuration

* local BOX address

* local public/private key

### Shared Configuration

* remote BOX addresses of other peers in the reserve

* shared secret

* minimum acceptable replication factor

* 'normal' event dispatching frequency

* warning time window
* has a global default as well as an override for each limitation

## Conflict Resolution
[conflict-resolution]: #conflict-resolution

Although it can be assumed the snapshot is already free from conflicts caused by concurrent updates initiated from a user, conflicts may still arise from disk corruption or other unforseen keeper errors.

The disputing keepers will take the appropriate steps to resolve the conflict. If an appropriate resolution can not be achieved, an event is raised.

## Events
The following events are dispatched for an administrative UI.

* limitation imminent

* limitation encountered

* keeper health
* memory
* disk I/O
* CPU usage
* disk corruption

* unresolved conflict

* unacceptable replication factor

* keeper added / removed

* network disruption

### Event Types
* normal
* warning
* failure

### Warnings

Events dispatched before a failure occurs based on a forecasting heuristic to determine how quickly a limit will be reached.

### Event Frequency

Normal events are dispatched periodically (based on a config param) for historical reporting and warnings|failures are dispatched immediately.

## Implementation
[reference-level-explanation]: #reference-level-explanation

### Components

The components that are needed for this use case are:

* data set keeper

* data set retriever

* event dispatcher

* file integrity monitor

### Event Dispatcher

A mechanism for dispatching events and queueing them to prevent loss if a peer goes down.

### File Integrity Monitoring

For detecting / handling disk corruption.

A full sweep of the file system is periodically done comparing the contents of chunks on disk with a source of truth.


### Network Architecture

A peer-peer architecture is used over master/slave so that if a single BOX goes down the rest of the reserve will still be able to operate normally.

* any shared config data is stored on each BOX

* any shared state required for the retrieval of a data set is stored on each BOX

* no central servers are used for networking


## Drawbacks
[drawbacks]: #drawbacks
Putting the responsibility of data reliability on BOX owners means there is potential for a BOX owner to make a mistake and permanently lose their data.

## Rationale and alternatives
[rationale-and-alternatives]: #rationale-and-alternatives
An alternative could be to use paid services (cloud storage providers) with their own SLAs to take on the responsibility of data reliability.

If participating in a decentralized storage network (DSN), the BOX owner could also purchase a mining component to offset their cost.

There are currently a few drawbacks with this:

* becoming a storage miner requires a significant upfront investment to cover hardware and staking costs

* a private reserve will always be more efficient since keepers will not have to worry about the overhead of trusting each other

These options are not mutually exclusive. Offering both options (free and paid) could provide the greatest freedom/flexibility for BOX owners.

## Dependencies

* [IPFS cluster](https://cluster.ipfs.io/)

* private connections

* authentication

* retriever

* keeper

@TODO - provide links to relevant docs

## Unresolved questions
[unresolved-questions]: #unresolved-questions

What is an ideal replication factor?

How can resilience be measured? Markov models?

How can contents of an entire filesystem be efficiently compared with a source of truth?

How can system resources and their limits be calculated?

How can NAT hole punching work in a pnet without any relays?

Should data compression be taken into account?

## Future Possibilities
[future-possibilities]: #future-possibilities

### History

Storing a history snapshots so an owner can go back in time and recover a data set from a previous state. This could help with data loss due to user errors.

### Regions

A person or group of people may have a large number of BOXes and want to set up geographically grouped regions such that if there is a severe network outage, each region will still be able to recover the entire contents of a snapshot.

, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
246 changes: 246 additions & 0 deletions docs/RFCs/personal-data-reserve.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
# Personal Data Reserve
- Start Date: 2022-03-18
- RFC PR: [functionland/docs/pull/61](https://github.com/functionland/docs/pull/61)
- Functionland Issue: [functionland/docs/issues/58](https://github.com/functionland/docs/issues/58)
- Status: Draft
- Authors: [Aaron Surty](https://github.com/gitaaron),
- Reviewers: [Farhoud](https://github.com/farhoud), [Ehsan](https://github.com/ehsan6sha), [Masih](https://github.com/orgs/functionland/people/masih)

## Summary
[summary]: #summary

This RFC covers how a BOX customer's data can be replicated across a group of BOXes physically separated from each other to help prevent unwanted loss.

## Use Case
[use-case]: #use-case

A person owns 3-5 BOXes (eg/ one in their home, one at their office and one at a friend's house).

Their basement floods bricking the one located in their home.

They are still able to access all of the data that was uploaded to their bricked home BOX from one of their other BOXes or a brand new BOX provisioned with factory settings.

The following scenarios are handled:

* adding a new BOX to the reserve

* removing a BOX from the reserve

* data becomes corrupted on a BOX


## Terminology
[terminology]: #terminology

| Name | Definition |
|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
| reserve | a group of keepers collaboratively working together to back up a snapshot |
| snapshot | a collection of data that represents the entire file system being backed up at a given moment in time |
| chunk | a portion of a data set |
| keeper | a BOX process responsible for backing up a file system to the reserve |
| retriever | a BOX process given the task of rebuilding a usable file system from a snapshot |
| author | the BOX where the backed up data set was created |
| peer | a member of the reserve |
| replication factor | how many keepers a chunk of data is stored on; a greater replication factor means greater resilience |
| data set | a file, directory, or data from a database |
| warning time window | how much time should be given to send an alert before an imminent limitation is encountered and a system failure occurs |



## Pre-conditions
[pre-conditions]: #pre-conditions

* the owner has already authenticated themselves with each BOX

* the fula-api is persisting data to a disk accessible by the keeper

* each BOX is already provisioned with the necessary configuration info required to operate a reserve

## Assumptions

* each keeper in a reserve can be trusted to not operate maliciously by performing unwanted delete operations

* the data being backed up is:
* multimedia files such as photos and video
* [orbit db](https://orbitdb.org/) metadata
* plain text files for shared configuration data

## Out of Scope

* high-frequency updates to data sets which might be encountered with multi-tenancy or recording streaming video

* syncing of data between BOXes so that it is usable on each BOX (eg/ CRUD operations on them in real time)

* conflicts arising from concurrent updates initiated by a user on several different BOXes

* heuristics used to minimize bandwidth and improve availability

## Limitations
[limitations]: #limitations

The following limitations may be encountered while operating a reserve:
* file size
* number of files in a directory
* snapshot size
* number of keepers in a reserve

## Configuration
[configuration]: #configuration

Configuration data for each peer can be split into local and shared.

### Local Configuration

* local BOX address

* local public/private key

### Shared Configuration

* remote BOX addresses of other peers in the reserve

* shared secret

* minimum acceptable replication factor

* 'normal' event dispatching frequency

* warning time window
* has a global default as well as an override for each limitation

## Conflict Resolution
[conflict-resolution]: #conflict-resolution

Although it can be assumed the snapshot is already free from conflicts caused by concurrent updates initiated from a user, conflicts may still arise from disk corruption or other unforseen keeper errors.

The disputing keepers will take the appropriate steps to resolve the conflict. If an appropriate resolution can not be achieved, an event is raised.

## Events
The following events are dispatched for an administrative UI.

* limitation imminent

* limitation encountered

* keeper health
* memory
* disk I/O
* CPU usage
* disk corruption

* unresolved conflict

* unacceptable replication factor

* keeper added / removed

* network disruption

### Event Types
* normal
* warning
* failure

### Warnings

Events dispatched before a failure occurs based on a forecasting heuristic to determine how quickly a limit will be reached.

### Event Frequency

Normal events are dispatched periodically (based on a config param) for historical reporting and warnings|failures are dispatched immediately.

## Implementation
[reference-level-explanation]: #reference-level-explanation

### Components

The components that are needed for this use case are:

* data set keeper

* data set retriever

* event dispatcher

* file integrity monitor

### Event Dispatcher

A mechanism for dispatching events and queueing them to prevent loss if a peer goes down.

### File Integrity Monitoring

For detecting / handling disk corruption.

A full sweep of the file system is periodically done comparing the contents of chunks on disk with a source of truth.


### Network Architecture

A peer-peer architecture is used over master/slave so that if a single BOX goes down the rest of the reserve will still be able to operate normally.

* any shared config data is stored on each BOX

* any shared state required for the retrieval of a data set is stored on each BOX

* no central servers are used for networking


## Drawbacks
[drawbacks]: #drawbacks
Putting the responsibility of data reliability on BOX owners means there is potential for a BOX owner to make a mistake and permanently lose their data.

## Rationale and alternatives
[rationale-and-alternatives]: #rationale-and-alternatives
An alternative could be to use paid services (cloud storage providers) with their own SLAs to take on the responsibility of data reliability.

If participating in a decentralized storage network (DSN), the BOX owner could also purchase a mining component to offset their cost.

There are currently a few drawbacks with this:

* becoming a storage miner requires a significant upfront investment to cover hardware and staking costs

* a private reserve will always be more efficient since keepers will not have to worry about the overhead of trusting each other

These options are not mutually exclusive. Offering both options (free and paid) could provide the greatest freedom/flexibility for BOX owners.

## Dependencies

* [IPFS cluster](https://cluster.ipfs.io/)

* private connections

* authentication

* retriever

* keeper

@TODO - provide links to relevant docs

## Unresolved questions
[unresolved-questions]: #unresolved-questions

What is an ideal replication factor?

How can resilience be measured? Markov models?

How can contents of an entire filesystem be efficiently compared with a source of truth?

How can system resources and their limits be calculated?

How can NAT hole punching work in a pnet without any relays?

Should data compression be taken into account?

## Future Possibilities
[future-possibilities]: #future-possibilities

### History

Storing a history snapshots so an owner can go back in time and recover a data set from a previous state. This could help with data loss due to user errors.

### Regions

A person or group of people may have a large number of BOXes and want to set up geographically grouped regions such that if there is a severe network outage, each region will still be able to recover the entire contents of a snapshot.

, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
246 changes: 246 additions & 0 deletions docs/RFCs/personal-data-reserve.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
# Personal Data Reserve
- Start Date: 2022-03-18
- RFC PR: [functionland/docs/pull/61](https://github.com/functionland/docs/pull/61)
- Functionland Issue: [functionland/docs/issues/58](https://github.com/functionland/docs/issues/58)
- Status: Draft
- Authors: [Aaron Surty](https://github.com/gitaaron),
- Reviewers: [Farhoud](https://github.com/farhoud), [Ehsan](https://github.com/ehsan6sha), [Masih](https://github.com/orgs/functionland/people/masih)

## Summary
[summary]: #summary

This RFC covers how a BOX customer's data can be replicated across a group of BOXes physically separated from each other to help prevent unwanted loss.

## Use Case
[use-case]: #use-case

A person owns 3-5 BOXes (eg/ one in their home, one at their office and one at a friend's house).

Their basement floods bricking the one located in their home.

They are still able to access all of the data that was uploaded to their bricked home BOX from one of their other BOXes or a brand new BOX provisioned with factory settings.

The following scenarios are handled:

* adding a new BOX to the reserve

* removing a BOX from the reserve

* data becomes corrupted on a BOX


## Terminology
[terminology]: #terminology

| Name | Definition |
|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
| reserve | a group of keepers collaboratively working together to back up a snapshot |
| snapshot | a collection of data that represents the entire file system being backed up at a given moment in time |
| chunk | a portion of a data set |
| keeper | a BOX process responsible for backing up a file system to the reserve |
| retriever | a BOX process given the task of rebuilding a usable file system from a snapshot |
| author | the BOX where the backed up data set was created |
| peer | a member of the reserve |
| replication factor | how many keepers a chunk of data is stored on; a greater replication factor means greater resilience |
| data set | a file, directory, or data from a database |
| warning time window | how much time should be given to send an alert before an imminent limitation is encountered and a system failure occurs |



## Pre-conditions
[pre-conditions]: #pre-conditions

* the owner has already authenticated themselves with each BOX

* the fula-api is persisting data to a disk accessible by the keeper

* each BOX is already provisioned with the necessary configuration info required to operate a reserve

## Assumptions

* each keeper in a reserve can be trusted to not operate maliciously by performing unwanted delete operations

* the data being backed up is:
* multimedia files such as photos and video
* [orbit db](https://orbitdb.org/) metadata
* plain text files for shared configuration data

## Out of Scope

* high-frequency updates to data sets which might be encountered with multi-tenancy or recording streaming video

* syncing of data between BOXes so that it is usable on each BOX (eg/ CRUD operations on them in real time)

* conflicts arising from concurrent updates initiated by a user on several different BOXes

* heuristics used to minimize bandwidth and improve availability

## Limitations
[limitations]: #limitations

The following limitations may be encountered while operating a reserve:
* file size
* number of files in a directory
* snapshot size
* number of keepers in a reserve

## Configuration
[configuration]: #configuration

Configuration data for each peer can be split into local and shared.

### Local Configuration

* local BOX address

* local public/private key

### Shared Configuration

* remote BOX addresses of other peers in the reserve

* shared secret

* minimum acceptable replication factor

* 'normal' event dispatching frequency

* warning time window
* has a global default as well as an override for each limitation

## Conflict Resolution
[conflict-resolution]: #conflict-resolution

Although it can be assumed the snapshot is already free from conflicts caused by concurrent updates initiated from a user, conflicts may still arise from disk corruption or other unforseen keeper errors.

The disputing keepers will take the appropriate steps to resolve the conflict. If an appropriate resolution can not be achieved, an event is raised.

## Events
The following events are dispatched for an administrative UI.

* limitation imminent

* limitation encountered

* keeper health
* memory
* disk I/O
* CPU usage
* disk corruption

* unresolved conflict

* unacceptable replication factor

* keeper added / removed

* network disruption

### Event Types
* normal
* warning
* failure

### Warnings

Events dispatched before a failure occurs based on a forecasting heuristic to determine how quickly a limit will be reached.

### Event Frequency

Normal events are dispatched periodically (based on a config param) for historical reporting and warnings|failures are dispatched immediately.

## Implementation
[reference-level-explanation]: #reference-level-explanation

### Components

The components that are needed for this use case are:

* data set keeper

* data set retriever

* event dispatcher

* file integrity monitor

### Event Dispatcher

A mechanism for dispatching events and queueing them to prevent loss if a peer goes down.

### File Integrity Monitoring

For detecting / handling disk corruption.

A full sweep of the file system is periodically done comparing the contents of chunks on disk with a source of truth.


### Network Architecture

A peer-peer architecture is used over master/slave so that if a single BOX goes down the rest of the reserve will still be able to operate normally.

* any shared config data is stored on each BOX

* any shared state required for the retrieval of a data set is stored on each BOX

* no central servers are used for networking


## Drawbacks
[drawbacks]: #drawbacks
Putting the responsibility of data reliability on BOX owners means there is potential for a BOX owner to make a mistake and permanently lose their data.

## Rationale and alternatives
[rationale-and-alternatives]: #rationale-and-alternatives
An alternative could be to use paid services (cloud storage providers) with their own SLAs to take on the responsibility of data reliability.

If participating in a decentralized storage network (DSN), the BOX owner could also purchase a mining component to offset their cost.

There are currently a few drawbacks with this:

* becoming a storage miner requires a significant upfront investment to cover hardware and staking costs

* a private reserve will always be more efficient since keepers will not have to worry about the overhead of trusting each other

These options are not mutually exclusive. Offering both options (free and paid) could provide the greatest freedom/flexibility for BOX owners.

## Dependencies

* [IPFS cluster](https://cluster.ipfs.io/)

* private connections

* authentication

* retriever

* keeper

@TODO - provide links to relevant docs

## Unresolved questions
[unresolved-questions]: #unresolved-questions

What is an ideal replication factor?

How can resilience be measured? Markov models?

How can contents of an entire filesystem be efficiently compared with a source of truth?

How can system resources and their limits be calculated?

How can NAT hole punching work in a pnet without any relays?

Should data compression be taken into account?

## Future Possibilities
[future-possibilities]: #future-possibilities

### History

Storing a history snapshots so an owner can go back in time and recover a data set from a previous state. This could help with data loss due to user errors.

### Regions

A person or group of people may have a large number of BOXes and want to set up geographically grouped regions such that if there is a severe network outage, each region will still be able to recover the entire contents of a snapshot.

, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
246 changes: 246 additions & 0 deletions docs/RFCs/personal-data-reserve.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
# Personal Data Reserve
- Start Date: 2022-03-18
- RFC PR: [functionland/docs/pull/61](https://github.com/functionland/docs/pull/61)
- Functionland Issue: [functionland/docs/issues/58](https://github.com/functionland/docs/issues/58)
- Status: Draft
- Authors: [Aaron Surty](https://github.com/gitaaron),
- Reviewers: [Farhoud](https://github.com/farhoud), [Ehsan](https://github.com/ehsan6sha), [Masih](https://github.com/orgs/functionland/people/masih)

## Summary
[summary]: #summary

This RFC covers how a BOX customer's data can be replicated across a group of BOXes physically separated from each other to help prevent unwanted loss.

## Use Case
[use-case]: #use-case

A person owns 3-5 BOXes (eg/ one in their home, one at their office and one at a friend's house).

Their basement floods bricking the one located in their home.

They are still able to access all of the data that was uploaded to their bricked home BOX from one of their other BOXes or a brand new BOX provisioned with factory settings.

The following scenarios are handled:

* adding a new BOX to the reserve

* removing a BOX from the reserve

* data becomes corrupted on a BOX


## Terminology
[terminology]: #terminology

| Name | Definition |
|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
| reserve | a group of keepers collaboratively working together to back up a snapshot |
| snapshot | a collection of data that represents the entire file system being backed up at a given moment in time |
| chunk | a portion of a data set |
| keeper | a BOX process responsible for backing up a file system to the reserve |
| retriever | a BOX process given the task of rebuilding a usable file system from a snapshot |
| author | the BOX where the backed up data set was created |
| peer | a member of the reserve |
| replication factor | how many keepers a chunk of data is stored on; a greater replication factor means greater resilience |
| data set | a file, directory, or data from a database |
| warning time window | how much time should be given to send an alert before an imminent limitation is encountered and a system failure occurs |



## Pre-conditions
[pre-conditions]: #pre-conditions

* the owner has already authenticated themselves with each BOX

* the fula-api is persisting data to a disk accessible by the keeper

* each BOX is already provisioned with the necessary configuration info required to operate a reserve

## Assumptions

* each keeper in a reserve can be trusted to not operate maliciously by performing unwanted delete operations

* the data being backed up is:
* multimedia files such as photos and video
* [orbit db](https://orbitdb.org/) metadata
* plain text files for shared configuration data

## Out of Scope

* high-frequency updates to data sets which might be encountered with multi-tenancy or recording streaming video

* syncing of data between BOXes so that it is usable on each BOX (eg/ CRUD operations on them in real time)

* conflicts arising from concurrent updates initiated by a user on several different BOXes

* heuristics used to minimize bandwidth and improve availability

## Limitations
[limitations]: #limitations

The following limitations may be encountered while operating a reserve:
* file size
* number of files in a directory
* snapshot size
* number of keepers in a reserve

## Configuration
[configuration]: #configuration

Configuration data for each peer can be split into local and shared.

### Local Configuration

* local BOX address

* local public/private key

### Shared Configuration

* remote BOX addresses of other peers in the reserve

* shared secret

* minimum acceptable replication factor

* 'normal' event dispatching frequency

* warning time window
* has a global default as well as an override for each limitation

## Conflict Resolution
[conflict-resolution]: #conflict-resolution

Although it can be assumed the snapshot is already free from conflicts caused by concurrent updates initiated from a user, conflicts may still arise from disk corruption or other unforseen keeper errors.

The disputing keepers will take the appropriate steps to resolve the conflict. If an appropriate resolution can not be achieved, an event is raised.

## Events
The following events are dispatched for an administrative UI.

* limitation imminent

* limitation encountered

* keeper health
* memory
* disk I/O
* CPU usage
* disk corruption

* unresolved conflict

* unacceptable replication factor

* keeper added / removed

* network disruption

### Event Types
* normal
* warning
* failure

### Warnings

Events dispatched before a failure occurs based on a forecasting heuristic to determine how quickly a limit will be reached.

### Event Frequency

Normal events are dispatched periodically (based on a config param) for historical reporting and warnings|failures are dispatched immediately.

## Implementation
[reference-level-explanation]: #reference-level-explanation

### Components

The components that are needed for this use case are:

* data set keeper

* data set retriever

* event dispatcher

* file integrity monitor

### Event Dispatcher

A mechanism for dispatching events and queueing them to prevent loss if a peer goes down.

### File Integrity Monitoring

For detecting / handling disk corruption.

A full sweep of the file system is periodically done comparing the contents of chunks on disk with a source of truth.


### Network Architecture

A peer-peer architecture is used over master/slave so that if a single BOX goes down the rest of the reserve will still be able to operate normally.

* any shared config data is stored on each BOX

* any shared state required for the retrieval of a data set is stored on each BOX

* no central servers are used for networking


## Drawbacks
[drawbacks]: #drawbacks
Putting the responsibility of data reliability on BOX owners means there is potential for a BOX owner to make a mistake and permanently lose their data.

## Rationale and alternatives
[rationale-and-alternatives]: #rationale-and-alternatives
An alternative could be to use paid services (cloud storage providers) with their own SLAs to take on the responsibility of data reliability.

If participating in a decentralized storage network (DSN), the BOX owner could also purchase a mining component to offset their cost.

There are currently a few drawbacks with this:

* becoming a storage miner requires a significant upfront investment to cover hardware and staking costs

* a private reserve will always be more efficient since keepers will not have to worry about the overhead of trusting each other

These options are not mutually exclusive. Offering both options (free and paid) could provide the greatest freedom/flexibility for BOX owners.

## Dependencies

* [IPFS cluster](https://cluster.ipfs.io/)

* private connections

* authentication

* retriever

* keeper

@TODO - provide links to relevant docs

## Unresolved questions
[unresolved-questions]: #unresolved-questions

What is an ideal replication factor?

How can resilience be measured? Markov models?

How can contents of an entire filesystem be efficiently compared with a source of truth?

How can system resources and their limits be calculated?

How can NAT hole punching work in a pnet without any relays?

Should data compression be taken into account?

## Future Possibilities
[future-possibilities]: #future-possibilities

### History

Storing a history snapshots so an owner can go back in time and recover a data set from a previous state. This could help with data loss due to user errors.

### Regions

A person or group of people may have a large number of BOXes and want to set up geographically grouped regions such that if there is a severe network outage, each region will still be able to recover the entire contents of a snapshot.

, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
246 changes: 246 additions & 0 deletions docs/RFCs/personal-data-reserve.md
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,246 @@
# Personal Data Reserve
- Start Date: 2022-03-18
- RFC PR: [functionland/docs/pull/61](https://github.com/functionland/docs/pull/61)
- Functionland Issue: [functionland/docs/issues/58](https://github.com/functionland/docs/issues/58)
- Status: Draft
- Authors: [Aaron Surty](https://github.com/gitaaron),
- Reviewers: [Farhoud](https://github.com/farhoud), [Ehsan](https://github.com/ehsan6sha), [Masih](https://github.com/orgs/functionland/people/masih)

## Summary
[summary]: #summary

This RFC covers how a BOX customer's data can be replicated across a group of BOXes physically separated from each other to help prevent unwanted loss.

## Use Case
[use-case]: #use-case

A person owns 3-5 BOXes (eg/ one in their home, one at their office and one at a friend's house).

Their basement floods bricking the one located in their home.

They are still able to access all of the data that was uploaded to their bricked home BOX from one of their other BOXes or a brand new BOX provisioned with factory settings.

The following scenarios are handled:

* adding a new BOX to the reserve

* removing a BOX from the reserve

* data becomes corrupted on a BOX


## Terminology
[terminology]: #terminology

| Name | Definition |
|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
| reserve | a group of keepers collaboratively working together to back up a snapshot |
| snapshot | a collection of data that represents the entire file system being backed up at a given moment in time |
| chunk | a portion of a data set |
| keeper | a BOX process responsible for backing up a file system to the reserve |
| retriever | a BOX process given the task of rebuilding a usable file system from a snapshot |
| author | the BOX where the backed up data set was created |
| peer | a member of the reserve |
| replication factor | how many keepers a chunk of data is stored on; a greater replication factor means greater resilience |
| data set | a file, directory, or data from a database |
| warning time window | how much time should be given to send an alert before an imminent limitation is encountered and a system failure occurs |



## Pre-conditions
[pre-conditions]: #pre-conditions

* the owner has already authenticated themselves with each BOX

* the fula-api is persisting data to a disk accessible by the keeper

* each BOX is already provisioned with the necessary configuration info required to operate a reserve

## Assumptions

* each keeper in a reserve can be trusted to not operate maliciously by performing unwanted delete operations

* the data being backed up is:
* multimedia files such as photos and video
* [orbit db](https://orbitdb.org/) metadata
* plain text files for shared configuration data

## Out of Scope

* high-frequency updates to data sets which might be encountered with multi-tenancy or recording streaming video

* syncing of data between BOXes so that it is usable on each BOX (eg/ CRUD operations on them in real time)

* conflicts arising from concurrent updates initiated by a user on several different BOXes

* heuristics used to minimize bandwidth and improve availability

## Limitations
[limitations]: #limitations

The following limitations may be encountered while operating a reserve:
* file size
* number of files in a directory
* snapshot size
* number of keepers in a reserve

## Configuration
[configuration]: #configuration

Configuration data for each peer can be split into local and shared.

### Local Configuration

* local BOX address

* local public/private key

### Shared Configuration

* remote BOX addresses of other peers in the reserve

* shared secret

* minimum acceptable replication factor

* 'normal' event dispatching frequency

* warning time window
* has a global default as well as an override for each limitation

## Conflict Resolution
[conflict-resolution]: #conflict-resolution

Although it can be assumed the snapshot is already free from conflicts caused by concurrent updates initiated from a user, conflicts may still arise from disk corruption or other unforseen keeper errors.

The disputing keepers will take the appropriate steps to resolve the conflict. If an appropriate resolution can not be achieved, an event is raised.

## Events
The following events are dispatched for an administrative UI.

* limitation imminent

* limitation encountered

* keeper health
* memory
* disk I/O
* CPU usage
* disk corruption

* unresolved conflict

* unacceptable replication factor

* keeper added / removed

* network disruption

### Event Types
* normal
* warning
* failure

### Warnings

Events dispatched before a failure occurs based on a forecasting heuristic to determine how quickly a limit will be reached.

### Event Frequency

Normal events are dispatched periodically (based on a config param) for historical reporting and warnings|failures are dispatched immediately.

## Implementation
[reference-level-explanation]: #reference-level-explanation

### Components

The components that are needed for this use case are:

* data set keeper

* data set retriever

* event dispatcher

* file integrity monitor

### Event Dispatcher

A mechanism for dispatching events and queueing them to prevent loss if a peer goes down.

### File Integrity Monitoring

For detecting / handling disk corruption.

A full sweep of the file system is periodically done comparing the contents of chunks on disk with a source of truth.


### Network Architecture

A peer-peer architecture is used over master/slave so that if a single BOX goes down the rest of the reserve will still be able to operate normally.

* any shared config data is stored on each BOX

* any shared state required for the retrieval of a data set is stored on each BOX

* no central servers are used for networking


## Drawbacks
[drawbacks]: #drawbacks
Putting the responsibility of data reliability on BOX owners means there is potential for a BOX owner to make a mistake and permanently lose their data.

## Rationale and alternatives
[rationale-and-alternatives]: #rationale-and-alternatives
An alternative could be to use paid services (cloud storage providers) with their own SLAs to take on the responsibility of data reliability.

If participating in a decentralized storage network (DSN), the BOX owner could also purchase a mining component to offset their cost.

There are currently a few drawbacks with this:

* becoming a storage miner requires a significant upfront investment to cover hardware and staking costs

* a private reserve will always be more efficient since keepers will not have to worry about the overhead of trusting each other

These options are not mutually exclusive. Offering both options (free and paid) could provide the greatest freedom/flexibility for BOX owners.

## Dependencies

* [IPFS cluster](https://cluster.ipfs.io/)

* private connections

* authentication

* retriever

* keeper

@TODO - provide links to relevant docs

## Unresolved questions
[unresolved-questions]: #unresolved-questions

What is an ideal replication factor?

How can resilience be measured? Markov models?

How can contents of an entire filesystem be efficiently compared with a source of truth?

How can system resources and their limits be calculated?

How can NAT hole punching work in a pnet without any relays?

Should data compression be taken into account?

## Future Possibilities
[future-possibilities]: #future-possibilities

### History

Storing a history snapshots so an owner can go back in time and recover a data set from a previous state. This could help with data loss due to user errors.

### Regions

A person or group of people may have a large number of BOXes and want to set up geographically grouped regions such that if there is a severe network outage, each region will still be able to recover the entire contents of a snapshot.