Home

Jump to bottomEditNew page
egaumer edited this page Jan 13, 2012 · 8 revisions

Pypes Documentation

If you're anything like us then you've probably written countless numbers of scripts to manipluate data in various way. The more operations/transformations on the data, the more your code starts to look like spaghetti. What if you could write small portions of logic that each perfrom a specific task and then connect them all together so that the output of one task becomes the input of another? This is precisely how Unix pipes work.

Pypes is an attempt to provide that same ideology in a more modern world. Pypes allows you to write small applications called "components". Each component focuses on providing a specific operation on a stream of data objects called "packets". This programming paradigm is referred to as flow-based programming and pypes supports a subset of flow-based features.

Pypes also provides a rich Web 2.0 interface that allows users to connect components using simple drag-n-drop functionality. The UI was highly inspired by Yahoo! Pipes and provides a similar experience. The end result is a visual programming experience where non-developers can build complex data flow architectures for manipulating digital content.

If you are a developer then pypes allows you to write custom components. The UI will detect any custom components and make them available for use. Pypes even provides project templates for generating new components. The templates create the boilerplate code necessary for writing custom components as well as the build scripts used to produce the final product. If you're familiar with Python, components are nothing more than egg files that define specific entry points.

Pypes is more than just a pretty face though. The underlying framework is capable of utilizing multi-core/cpu architectures and avoids the overhead associated with traditional thread models. Each component is an abstraction of a Stackless tasklet and consumes only a few hundred bytes of memory. Switching context between components is extremely robust allowing systems to scale up and take full advantage of the underlying hardware.

At the same time, the system scales out through its REST interface. For large data streams, more nodes can be added to build out entire clusters capable of processing a large number of documents in parallel. The system is completely decentralized (no single point of failure) and homogonous (working separately to achieve a common goal).

Lastly, pypes is not limited to linear pipeline models. It provides full support for directed acyclic graphs (DAG) giving you the ability to publish content streams to multiple end points and aggregate several input streams with branching and merging operations.

Our original intent when we started developing pypes was to build a framework for indexing content for search. This typically involves hundreds of millions of documents having to be scrubbed, transformed, and indexed. This played a major role in the design decisions and pushed us to create something that was highly scalable. To that extent, the system also scales down nicely allowing it to run on very modest hardware.

Whenever we introduce someone new to pypes we always hear about interesting ideas we never envisioned. We've come to realize that the system has the potential to solve a number of interesting problems that were never part of the original roadmap. If you have any interesting ideas we'd like to hear about them.

Enjoy!
The pypes development team

Features

  • RESTful HTTP Interface (Web 2.0)
  • Pure Message Passing (JSON objects)
  • Distributed (Decentralized & Homogonous)
  • Micro-Threaded (Components are Stackless tasklets)
  • Multi-processor support (Utilize SMP and multicore hardware)
  • Lightweight (Scales down)
  • Pluggable Architecure (Create your own components)
  • Project Templates (Generate boilerplate components)
  • Simple Authentication (Password protection)
  • Import/Export Projects (Share your project)

Requirements

Documentation

Contact Information

Committers

Add a custom footer
Add a custom sidebar

Clone this wiki locally

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all 
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Home
Jump to bottomEditNew page

egaumer edited this page Jan 13, 2012
·
8 revisions
Pypes Documentation
If you're anything like us then you've probably written countless numbers of scripts to manipluate data in various way. The more operations/transformations on the data, the more your code starts to look like spaghetti. What if you could write small portions of logic that each perfrom a specific task and then connect them all together so that the output of one task becomes the input of another? This is precisely how Unix pipes work.
Pypes is an attempt to provide that same ideology in a more modern world. Pypes allows you to write small applications called "components". Each component focuses on providing a specific operation on a stream of data objects called "packets". This programming paradigm is referred to as flow-based programming and pypes supports a subset of flow-based features.
Pypes also provides a rich Web 2.0 interface that allows users to connect components using simple drag-n-drop functionality. The UI was highly inspired by Yahoo! Pipes and provides a similar experience. The end result is a visual programming experience where non-developers can build complex data flow architectures for manipulating digital content.
If you are a developer then pypes allows you to write custom components. The UI will detect any custom components and make them available for use. Pypes even provides project templates for generating new components. The templates create the boilerplate code necessary for writing custom components as well as the build scripts used to produce the final product. If you're familiar with Python, components are nothing more than egg files that define specific entry points.
Pypes is more than just a pretty face though. The underlying framework is capable of utilizing multi-core/cpu architectures and avoids the overhead associated with traditional thread models. Each component is an abstraction of a Stackless tasklet and consumes only a few hundred bytes of memory. Switching context between components is extremely robust allowing systems to scale up and take full advantage of the underlying hardware.
At the same time, the system scales out through its REST interface. For large data streams, more nodes can be added to build out entire clusters capable of processing a large number of documents in parallel. The system is completely decentralized (no single point of failure) and homogonous (working separately to achieve a common goal).
Lastly, pypes is not limited to linear pipeline models. It provides full support for directed acyclic graphs (DAG) giving you the ability to publish content streams to multiple end points and aggregate several input streams with branching and merging operations.
Our original intent when we started developing pypes was to build a framework for indexing content for search. This typically involves hundreds of millions of documents having to be scrubbed, transformed, and indexed. This played a major role in the design decisions and pushed us to create something that was highly scalable. To that extent, the system also scales down nicely allowing it to run on very modest hardware.
Whenever we introduce someone new to pypes we always hear about interesting ideas we never envisioned. We've come to realize that the system has the potential to solve a number of interesting problems that were never part of the original roadmap. If you have any interesting ideas we'd like to hear about them.
Enjoy! 
 The pypes development team
Features
  • RESTful HTTP Interface (Web 2.0)
  • Pure Message Passing (JSON objects)
  • Distributed (Decentralized & Homogonous)
  • Micro-Threaded (Components are Stackless tasklets)
  • Multi-processor support (Utilize SMP and multicore hardware)
  • Lightweight (Scales down)
  • Pluggable Architecure (Create your own components)
  • Project Templates (Generate boilerplate components)
  • Simple Authentication (Password protection)
  • Import/Export Projects (Share your project) 
Requirements
Documentation
Contact Information
Committers
 Add a custom footer

 Add a custom sidebar

Clone this wiki locally

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix");
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Home
Jump to bottomEditNew page

egaumer edited this page Jan 13, 2012
·
8 revisions
Pypes Documentation
If you're anything like us then you've probably written countless numbers of scripts to manipluate data in various way. The more operations/transformations on the data, the more your code starts to look like spaghetti. What if you could write small portions of logic that each perfrom a specific task and then connect them all together so that the output of one task becomes the input of another? This is precisely how Unix pipes work.
Pypes is an attempt to provide that same ideology in a more modern world. Pypes allows you to write small applications called "components". Each component focuses on providing a specific operation on a stream of data objects called "packets". This programming paradigm is referred to as flow-based programming and pypes supports a subset of flow-based features.
Pypes also provides a rich Web 2.0 interface that allows users to connect components using simple drag-n-drop functionality. The UI was highly inspired by Yahoo! Pipes and provides a similar experience. The end result is a visual programming experience where non-developers can build complex data flow architectures for manipulating digital content.
If you are a developer then pypes allows you to write custom components. The UI will detect any custom components and make them available for use. Pypes even provides project templates for generating new components. The templates create the boilerplate code necessary for writing custom components as well as the build scripts used to produce the final product. If you're familiar with Python, components are nothing more than egg files that define specific entry points.
Pypes is more than just a pretty face though. The underlying framework is capable of utilizing multi-core/cpu architectures and avoids the overhead associated with traditional thread models. Each component is an abstraction of a Stackless tasklet and consumes only a few hundred bytes of memory. Switching context between components is extremely robust allowing systems to scale up and take full advantage of the underlying hardware.
At the same time, the system scales out through its REST interface. For large data streams, more nodes can be added to build out entire clusters capable of processing a large number of documents in parallel. The system is completely decentralized (no single point of failure) and homogonous (working separately to achieve a common goal).
Lastly, pypes is not limited to linear pipeline models. It provides full support for directed acyclic graphs (DAG) giving you the ability to publish content streams to multiple end points and aggregate several input streams with branching and merging operations.
Our original intent when we started developing pypes was to build a framework for indexing content for search. This typically involves hundreds of millions of documents having to be scrubbed, transformed, and indexed. This played a major role in the design decisions and pushed us to create something that was highly scalable. To that extent, the system also scales down nicely allowing it to run on very modest hardware.
Whenever we introduce someone new to pypes we always hear about interesting ideas we never envisioned. We've come to realize that the system has the potential to solve a number of interesting problems that were never part of the original roadmap. If you have any interesting ideas we'd like to hear about them.
Enjoy! 
 The pypes development team
Features
  • RESTful HTTP Interface (Web 2.0)
  • Pure Message Passing (JSON objects)
  • Distributed (Decentralized & Homogonous)
  • Micro-Threaded (Components are Stackless tasklets)
  • Multi-processor support (Utilize SMP and multicore hardware)
  • Lightweight (Scales down)
  • Pluggable Architecure (Create your own components)
  • Project Templates (Generate boilerplate components)
  • Simple Authentication (Password protection)
  • Import/Export Projects (Share your project) 
Requirements
Documentation
Contact Information
Committers
 Add a custom footer

 Add a custom sidebar

Clone this wiki locally

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms");
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Home
Jump to bottomEditNew page

egaumer edited this page Jan 13, 2012
·
8 revisions
Pypes Documentation
If you're anything like us then you've probably written countless numbers of scripts to manipluate data in various way. The more operations/transformations on the data, the more your code starts to look like spaghetti. What if you could write small portions of logic that each perfrom a specific task and then connect them all together so that the output of one task becomes the input of another? This is precisely how Unix pipes work.
Pypes is an attempt to provide that same ideology in a more modern world. Pypes allows you to write small applications called "components". Each component focuses on providing a specific operation on a stream of data objects called "packets". This programming paradigm is referred to as flow-based programming and pypes supports a subset of flow-based features.
Pypes also provides a rich Web 2.0 interface that allows users to connect components using simple drag-n-drop functionality. The UI was highly inspired by Yahoo! Pipes and provides a similar experience. The end result is a visual programming experience where non-developers can build complex data flow architectures for manipulating digital content.
If you are a developer then pypes allows you to write custom components. The UI will detect any custom components and make them available for use. Pypes even provides project templates for generating new components. The templates create the boilerplate code necessary for writing custom components as well as the build scripts used to produce the final product. If you're familiar with Python, components are nothing more than egg files that define specific entry points.
Pypes is more than just a pretty face though. The underlying framework is capable of utilizing multi-core/cpu architectures and avoids the overhead associated with traditional thread models. Each component is an abstraction of a Stackless tasklet and consumes only a few hundred bytes of memory. Switching context between components is extremely robust allowing systems to scale up and take full advantage of the underlying hardware.
At the same time, the system scales out through its REST interface. For large data streams, more nodes can be added to build out entire clusters capable of processing a large number of documents in parallel. The system is completely decentralized (no single point of failure) and homogonous (working separately to achieve a common goal).
Lastly, pypes is not limited to linear pipeline models. It provides full support for directed acyclic graphs (DAG) giving you the ability to publish content streams to multiple end points and aggregate several input streams with branching and merging operations.
Our original intent when we started developing pypes was to build a framework for indexing content for search. This typically involves hundreds of millions of documents having to be scrubbed, transformed, and indexed. This played a major role in the design decisions and pushed us to create something that was highly scalable. To that extent, the system also scales down nicely allowing it to run on very modest hardware.
Whenever we introduce someone new to pypes we always hear about interesting ideas we never envisioned. We've come to realize that the system has the potential to solve a number of interesting problems that were never part of the original roadmap. If you have any interesting ideas we'd like to hear about them.
Enjoy! 
 The pypes development team
Features
  • RESTful HTTP Interface (Web 2.0)
  • Pure Message Passing (JSON objects)
  • Distributed (Decentralized & Homogonous)
  • Micro-Threaded (Components are Stackless tasklets)
  • Multi-processor support (Utilize SMP and multicore hardware)
  • Lightweight (Scales down)
  • Pluggable Architecure (Create your own components)
  • Project Templates (Generate boilerplate components)
  • Simple Authentication (Password protection)
  • Import/Export Projects (Share your project) 
Requirements
Documentation
Contact Information
Committers
 Add a custom footer

 Add a custom sidebar

Clone this wiki locally

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links");
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Home
Jump to bottomEditNew page

egaumer edited this page Jan 13, 2012
·
8 revisions
Pypes Documentation
If you're anything like us then you've probably written countless numbers of scripts to manipluate data in various way. The more operations/transformations on the data, the more your code starts to look like spaghetti. What if you could write small portions of logic that each perfrom a specific task and then connect them all together so that the output of one task becomes the input of another? This is precisely how Unix pipes work.
Pypes is an attempt to provide that same ideology in a more modern world. Pypes allows you to write small applications called "components". Each component focuses on providing a specific operation on a stream of data objects called "packets". This programming paradigm is referred to as flow-based programming and pypes supports a subset of flow-based features.
Pypes also provides a rich Web 2.0 interface that allows users to connect components using simple drag-n-drop functionality. The UI was highly inspired by Yahoo! Pipes and provides a similar experience. The end result is a visual programming experience where non-developers can build complex data flow architectures for manipulating digital content.
If you are a developer then pypes allows you to write custom components. The UI will detect any custom components and make them available for use. Pypes even provides project templates for generating new components. The templates create the boilerplate code necessary for writing custom components as well as the build scripts used to produce the final product. If you're familiar with Python, components are nothing more than egg files that define specific entry points.
Pypes is more than just a pretty face though. The underlying framework is capable of utilizing multi-core/cpu architectures and avoids the overhead associated with traditional thread models. Each component is an abstraction of a Stackless tasklet and consumes only a few hundred bytes of memory. Switching context between components is extremely robust allowing systems to scale up and take full advantage of the underlying hardware.
At the same time, the system scales out through its REST interface. For large data streams, more nodes can be added to build out entire clusters capable of processing a large number of documents in parallel. The system is completely decentralized (no single point of failure) and homogonous (working separately to achieve a common goal).
Lastly, pypes is not limited to linear pipeline models. It provides full support for directed acyclic graphs (DAG) giving you the ability to publish content streams to multiple end points and aggregate several input streams with branching and merging operations.
Our original intent when we started developing pypes was to build a framework for indexing content for search. This typically involves hundreds of millions of documents having to be scrubbed, transformed, and indexed. This played a major role in the design decisions and pushed us to create something that was highly scalable. To that extent, the system also scales down nicely allowing it to run on very modest hardware.
Whenever we introduce someone new to pypes we always hear about interesting ideas we never envisioned. We've come to realize that the system has the potential to solve a number of interesting problems that were never part of the original roadmap. If you have any interesting ideas we'd like to hear about them.
Enjoy! 
 The pypes development team
Features
  • RESTful HTTP Interface (Web 2.0)
  • Pure Message Passing (JSON objects)
  • Distributed (Decentralized & Homogonous)
  • Micro-Threaded (Components are Stackless tasklets)
  • Multi-processor support (Utilize SMP and multicore hardware)
  • Lightweight (Scales down)
  • Pluggable Architecure (Create your own components)
  • Project Templates (Generate boilerplate components)
  • Simple Authentication (Password protection)
  • Import/Export Projects (Share your project) 
Requirements
Documentation
Contact Information
Committers
 Add a custom footer

 Add a custom sidebar

Clone this wiki locally

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default");
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Home
Jump to bottomEditNew page

egaumer edited this page Jan 13, 2012
·
8 revisions
Pypes Documentation
If you're anything like us then you've probably written countless numbers of scripts to manipluate data in various way. The more operations/transformations on the data, the more your code starts to look like spaghetti. What if you could write small portions of logic that each perfrom a specific task and then connect them all together so that the output of one task becomes the input of another? This is precisely how Unix pipes work.
Pypes is an attempt to provide that same ideology in a more modern world. Pypes allows you to write small applications called "components". Each component focuses on providing a specific operation on a stream of data objects called "packets". This programming paradigm is referred to as flow-based programming and pypes supports a subset of flow-based features.
Pypes also provides a rich Web 2.0 interface that allows users to connect components using simple drag-n-drop functionality. The UI was highly inspired by Yahoo! Pipes and provides a similar experience. The end result is a visual programming experience where non-developers can build complex data flow architectures for manipulating digital content.
If you are a developer then pypes allows you to write custom components. The UI will detect any custom components and make them available for use. Pypes even provides project templates for generating new components. The templates create the boilerplate code necessary for writing custom components as well as the build scripts used to produce the final product. If you're familiar with Python, components are nothing more than egg files that define specific entry points.
Pypes is more than just a pretty face though. The underlying framework is capable of utilizing multi-core/cpu architectures and avoids the overhead associated with traditional thread models. Each component is an abstraction of a Stackless tasklet and consumes only a few hundred bytes of memory. Switching context between components is extremely robust allowing systems to scale up and take full advantage of the underlying hardware.
At the same time, the system scales out through its REST interface. For large data streams, more nodes can be added to build out entire clusters capable of processing a large number of documents in parallel. The system is completely decentralized (no single point of failure) and homogonous (working separately to achieve a common goal).
Lastly, pypes is not limited to linear pipeline models. It provides full support for directed acyclic graphs (DAG) giving you the ability to publish content streams to multiple end points and aggregate several input streams with branching and merging operations.
Our original intent when we started developing pypes was to build a framework for indexing content for search. This typically involves hundreds of millions of documents having to be scrubbed, transformed, and indexed. This played a major role in the design decisions and pushed us to create something that was highly scalable. To that extent, the system also scales down nicely allowing it to run on very modest hardware.
Whenever we introduce someone new to pypes we always hear about interesting ideas we never envisioned. We've come to realize that the system has the potential to solve a number of interesting problems that were never part of the original roadmap. If you have any interesting ideas we'd like to hear about them.
Enjoy! 
 The pypes development team
Features
  • RESTful HTTP Interface (Web 2.0)
  • Pure Message Passing (JSON objects)
  • Distributed (Decentralized & Homogonous)
  • Micro-Threaded (Components are Stackless tasklets)
  • Multi-processor support (Utilize SMP and multicore hardware)
  • Lightweight (Scales down)
  • Pluggable Architecure (Create your own components)
  • Project Templates (Generate boilerplate components)
  • Simple Authentication (Password protection)
  • Import/Export Projects (Share your project) 
Requirements
Documentation
Contact Information
Committers
 Add a custom footer

 Add a custom sidebar

Clone this wiki locally

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers");
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Home
Jump to bottomEditNew page

egaumer edited this page Jan 13, 2012
·
8 revisions
Pypes Documentation
If you're anything like us then you've probably written countless numbers of scripts to manipluate data in various way. The more operations/transformations on the data, the more your code starts to look like spaghetti. What if you could write small portions of logic that each perfrom a specific task and then connect them all together so that the output of one task becomes the input of another? This is precisely how Unix pipes work.
Pypes is an attempt to provide that same ideology in a more modern world. Pypes allows you to write small applications called "components". Each component focuses on providing a specific operation on a stream of data objects called "packets". This programming paradigm is referred to as flow-based programming and pypes supports a subset of flow-based features.
Pypes also provides a rich Web 2.0 interface that allows users to connect components using simple drag-n-drop functionality. The UI was highly inspired by Yahoo! Pipes and provides a similar experience. The end result is a visual programming experience where non-developers can build complex data flow architectures for manipulating digital content.
If you are a developer then pypes allows you to write custom components. The UI will detect any custom components and make them available for use. Pypes even provides project templates for generating new components. The templates create the boilerplate code necessary for writing custom components as well as the build scripts used to produce the final product. If you're familiar with Python, components are nothing more than egg files that define specific entry points.
Pypes is more than just a pretty face though. The underlying framework is capable of utilizing multi-core/cpu architectures and avoids the overhead associated with traditional thread models. Each component is an abstraction of a Stackless tasklet and consumes only a few hundred bytes of memory. Switching context between components is extremely robust allowing systems to scale up and take full advantage of the underlying hardware.
At the same time, the system scales out through its REST interface. For large data streams, more nodes can be added to build out entire clusters capable of processing a large number of documents in parallel. The system is completely decentralized (no single point of failure) and homogonous (working separately to achieve a common goal).
Lastly, pypes is not limited to linear pipeline models. It provides full support for directed acyclic graphs (DAG) giving you the ability to publish content streams to multiple end points and aggregate several input streams with branching and merging operations.
Our original intent when we started developing pypes was to build a framework for indexing content for search. This typically involves hundreds of millions of documents having to be scrubbed, transformed, and indexed. This played a major role in the design decisions and pushed us to create something that was highly scalable. To that extent, the system also scales down nicely allowing it to run on very modest hardware.
Whenever we introduce someone new to pypes we always hear about interesting ideas we never envisioned. We've come to realize that the system has the potential to solve a number of interesting problems that were never part of the original roadmap. If you have any interesting ideas we'd like to hear about them.
Enjoy! 
 The pypes development team
Features
  • RESTful HTTP Interface (Web 2.0)
  • Pure Message Passing (JSON objects)
  • Distributed (Decentralized & Homogonous)
  • Micro-Threaded (Components are Stackless tasklets)
  • Multi-processor support (Utilize SMP and multicore hardware)
  • Lightweight (Scales down)
  • Pluggable Architecure (Create your own components)
  • Project Templates (Generate boilerplate components)
  • Simple Authentication (Password protection)
  • Import/Export Projects (Share your project) 
Requirements
Documentation
Contact Information
Committers
 Add a custom footer

 Add a custom sidebar

Clone this wiki locally

, 'i');
if (__m === '*' || __re.test(location.href)) {
injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode");
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Home
Jump to bottomEditNew page

egaumer edited this page Jan 13, 2012
·
8 revisions
Pypes Documentation
If you're anything like us then you've probably written countless numbers of scripts to manipluate data in various way. The more operations/transformations on the data, the more your code starts to look like spaghetti. What if you could write small portions of logic that each perfrom a specific task and then connect them all together so that the output of one task becomes the input of another? This is precisely how Unix pipes work.
Pypes is an attempt to provide that same ideology in a more modern world. Pypes allows you to write small applications called "components". Each component focuses on providing a specific operation on a stream of data objects called "packets". This programming paradigm is referred to as flow-based programming and pypes supports a subset of flow-based features.
Pypes also provides a rich Web 2.0 interface that allows users to connect components using simple drag-n-drop functionality. The UI was highly inspired by Yahoo! Pipes and provides a similar experience. The end result is a visual programming experience where non-developers can build complex data flow architectures for manipulating digital content.
If you are a developer then pypes allows you to write custom components. The UI will detect any custom components and make them available for use. Pypes even provides project templates for generating new components. The templates create the boilerplate code necessary for writing custom components as well as the build scripts used to produce the final product. If you're familiar with Python, components are nothing more than egg files that define specific entry points.
Pypes is more than just a pretty face though. The underlying framework is capable of utilizing multi-core/cpu architectures and avoids the overhead associated with traditional thread models. Each component is an abstraction of a Stackless tasklet and consumes only a few hundred bytes of memory. Switching context between components is extremely robust allowing systems to scale up and take full advantage of the underlying hardware.
At the same time, the system scales out through its REST interface. For large data streams, more nodes can be added to build out entire clusters capable of processing a large number of documents in parallel. The system is completely decentralized (no single point of failure) and homogonous (working separately to achieve a common goal).
Lastly, pypes is not limited to linear pipeline models. It provides full support for directed acyclic graphs (DAG) giving you the ability to publish content streams to multiple end points and aggregate several input streams with branching and merging operations.
Our original intent when we started developing pypes was to build a framework for indexing content for search. This typically involves hundreds of millions of documents having to be scrubbed, transformed, and indexed. This played a major role in the design decisions and pushed us to create something that was highly scalable. To that extent, the system also scales down nicely allowing it to run on very modest hardware.
Whenever we introduce someone new to pypes we always hear about interesting ideas we never envisioned. We've come to realize that the system has the potential to solve a number of interesting problems that were never part of the original roadmap. If you have any interesting ideas we'd like to hear about them.
Enjoy! 
 The pypes development team
Features
  • RESTful HTTP Interface (Web 2.0)
  • Pure Message Passing (JSON objects)
  • Distributed (Decentralized & Homogonous)
  • Micro-Threaded (Components are Stackless tasklets)
  • Multi-processor support (Utilize SMP and multicore hardware)
  • Lightweight (Scales down)
  • Pluggable Architecure (Create your own components)
  • Project Templates (Generate boilerplate components)
  • Simple Authentication (Password protection)
  • Import/Export Projects (Share your project) 
Requirements
Documentation
Contact Information
Committers
 Add a custom footer

 Add a custom sidebar

Clone this wiki locally