Skip to content
BranchesTags

Repository files navigation

PipeDreams - CSV Data Explorer

PipeDreams Header

PipeDreams is a data exploration and visualization tool designed to be simple, flexible, and powerful. Upload any CSV file, perform basic ETL transformations, visualize your data, and gain insights with built-in machine learning—all from a user-friendly Streamlit interface. If no file is uploaded, a default dataset (customers-100000.csv) is used for demonstration.

Features

  • CSV Upload: Upload any CSV file for immediate analysis and visualization.
  • Default Dataset: If no file is uploaded, the app loads a sample dataset (customers-100000.csv) located in the data/ directory.
  • ETL Transformations: Clean and transform data, remove missing values, and auto-convert data types.
  • Data Visualization: Interactive charts (scatter, bar, line, histogram, and box plots) to gain insights from your data.
  • Clustering Analysis: Use KMeans clustering to identify natural groupings within the data, helping to segment and classify.
  • Predictive Analysis: A synthetic column (Annual Purchase Amount) is included for testing linear regression, allowing users to explore predictive analysis features.

Getting Started with Docker

You can run PipeDreams using Docker to avoid setting up dependencies locally. The pre-built Docker image is available on DockerHub.

Pulling the Docker Image

Pull the latest Docker image from DockerHub:

docker pull anuclei/pipedreams:latest

Running the Docker Container

Run the application with Docker, exposing it on port 8501:

docker run -p 8501:8501 anuclei/pipedreams:latest

Once the container is running, open your browser and go to http://localhost:8501 to access the application.

Kubernetes Deployment

PipeDreams can also be deployed on a Kubernetes cluster. This deployment scenario uses Minikube for local Kubernetes clusters and includes configurations for high availability and autoscaling.

For detailed instructions and YAML configurations, refer to the Kubernetes Deployment Guide in the k8s directory.

Manual Installation

If you prefer not to use Docker, you can set up the app manually.

Prerequisites

  • Python (version 3.6 or higher)

Installation

  1. Clone the repository:

    git clone https://github.com/markjacksonfishing/pipedreams.git
cd pipedreams

  2. Set up a virtual environment:

    • MacOS/Linux: 
      python3 -m venv venv
source venv/bin/activate
    • Windows: 
      python -m venv venv
.\venv\Scripts\activate

  3. Install dependencies:

    pip install -r requirements.txt

  4. Run the application:

    • MacOS/Linux: 
      source venv/bin/activate
streamlit run app.py
    • Windows: 
      .\venv\Scripts\activate
streamlit run app.py

    The application will open in your default web browser at http://localhost:8501 and will look like this: PipeDreams Browser

  5. Deactivate the virtual environment (when finished):

    deactivate

How to Use

  1. Start the application by following the setup steps above (or run it via Docker).
  2. Upload a CSV file using the file uploader in the app, or view the default dataset if no file is uploaded.
  3. Explore the data with built-in ETL transformations and interactive visualizations.
  4. Perform clustering analysis and predictive analysis on available data.

Advanced Insights: Clustering and Predictive Analysis

  • Clustering Analysis: Select features for clustering, and the app will automatically group data into clusters using KMeans. This can reveal natural groupings in the data, such as customer segments.
  • Predictive Analysis: Select features and a target variable (e.g., the synthetic Annual Purchase Amount) for linear regression. The app will generate a prediction model, display a mean squared error metric, and show an interactive scatter plot comparing actual vs. predicted values.

Default Dataset: customers-100000.csv

The default dataset, customers-100000.csv, is located in the data/ directory. If no CSV file is uploaded, this dataset will automatically load, allowing users to test the ETL transformations, visualizations, clustering, and predictive analysis features without needing their own data file.

About

A play on pipelines, with a focus on making data accessible and insightful.

www.anuclei.com

Topics

backenddata-engineeringdata-processingdata-visualizationdeploymentetlfrontendmachine-learningpythonstreamlit

Resources

Readme
MIT license
Activity

Stars

0 stars

Watchers

1 watching

Forks

0 forks
Report repository

Releases

Packages

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
GitHub - markjacksonfishing/pipedreams: A play on pipelines, with a focus on making data accessible and insightful. · GitHub
Skip to content
BranchesTags
Repository files navigation
PipeDreams - CSV Data Explorer
PipeDreams Header
PipeDreams is a data exploration and visualization tool designed to be simple, flexible, and powerful. Upload any CSV file, perform basic ETL transformations, visualize your data, and gain insights with built-in machine learning—all from a user-friendly Streamlit interface. If no file is uploaded, a default dataset (customers-100000.csv) is used for demonstration.
Features
  • CSV Upload: Upload any CSV file for immediate analysis and visualization.
  • Default Dataset: If no file is uploaded, the app loads a sample dataset (customers-100000.csv) located in the data/ directory.
  • ETL Transformations: Clean and transform data, remove missing values, and auto-convert data types.
  • Data Visualization: Interactive charts (scatter, bar, line, histogram, and box plots) to gain insights from your data.
  • Clustering Analysis: Use KMeans clustering to identify natural groupings within the data, helping to segment and classify.
  • Predictive Analysis: A synthetic column (Annual Purchase Amount) is included for testing linear regression, allowing users to explore predictive analysis features.
Getting Started with Docker
You can run PipeDreams using Docker to avoid setting up dependencies locally. The pre-built Docker image is available on DockerHub.
Pulling the Docker Image
Pull the latest Docker image from DockerHub:
docker pull anuclei/pipedreams:latest
Running the Docker Container
Run the application with Docker, exposing it on port 8501:
docker run -p 8501:8501 anuclei/pipedreams:latest
Once the container is running, open your browser and go to http://localhost:8501 to access the application.
Kubernetes Deployment
PipeDreams can also be deployed on a Kubernetes cluster. This deployment scenario uses Minikube for local Kubernetes clusters and includes configurations for high availability and autoscaling.
For detailed instructions and YAML configurations, refer to the Kubernetes Deployment Guide in the k8s directory.
Manual Installation
If you prefer not to use Docker, you can set up the app manually.
Prerequisites
  • Python (version 3.6 or higher)
Installation
  1. Clone the repository:
    git clone https://github.com/markjacksonfishing/pipedreams.git
cd pipedreams
  2. Set up a virtual environment:
    • MacOS/Linux:

      python3 -m venv venv
source venv/bin/activate
    • Windows:

      python -m venv venv
.\venv\Scripts\activate
  3. Install dependencies:
    pip install -r requirements.txt
  4. Run the application:
    • MacOS/Linux:

      source venv/bin/activate
streamlit run app.py
    • Windows:

      .\venv\Scripts\activate
streamlit run app.py
    The application will open in your default web browser at http://localhost:8501 and will look like this:
PipeDreams Browser
  5. Deactivate the virtual environment (when finished):
    deactivate
How to Use
  1. Start the application by following the setup steps above (or run it via Docker).
  2. Upload a CSV file using the file uploader in the app, or view the default dataset if no file is uploaded.
  3. Explore the data with built-in ETL transformations and interactive visualizations.
  4. Perform clustering analysis and predictive analysis on available data.
Advanced Insights: Clustering and Predictive Analysis
  • Clustering Analysis: Select features for clustering, and the app will automatically group data into clusters using KMeans. This can reveal natural groupings in the data, such as customer segments.
  • Predictive Analysis: Select features and a target variable (e.g., the synthetic Annual Purchase Amount) for linear regression. The app will generate a prediction model, display a mean squared error metric, and show an interactive scatter plot comparing actual vs. predicted values.
Default Dataset: customers-100000.csv
The default dataset, customers-100000.csv, is located in the data/ directory. If no CSV file is uploaded, this dataset will automatically load, allowing users to test the ETL transformations, visualizations, clustering, and predictive analysis features without needing their own data file.
About
A play on pipelines, with a focus on making data accessible and insightful.
www.anuclei.com
Topics
backenddata-engineeringdata-processingdata-visualizationdeploymentetlfrontendmachine-learningpythonstreamlit
Resources
Readme
MIT license
Activity
Stars
0 stars
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - markjacksonfishing/pipedreams: A play on pipelines, with a focus on making data accessible and insightful. · GitHub
Skip to content
BranchesTags
Repository files navigation
PipeDreams - CSV Data Explorer
PipeDreams Header
PipeDreams is a data exploration and visualization tool designed to be simple, flexible, and powerful. Upload any CSV file, perform basic ETL transformations, visualize your data, and gain insights with built-in machine learning—all from a user-friendly Streamlit interface. If no file is uploaded, a default dataset (customers-100000.csv) is used for demonstration.
Features
  • CSV Upload: Upload any CSV file for immediate analysis and visualization.
  • Default Dataset: If no file is uploaded, the app loads a sample dataset (customers-100000.csv) located in the data/ directory.
  • ETL Transformations: Clean and transform data, remove missing values, and auto-convert data types.
  • Data Visualization: Interactive charts (scatter, bar, line, histogram, and box plots) to gain insights from your data.
  • Clustering Analysis: Use KMeans clustering to identify natural groupings within the data, helping to segment and classify.
  • Predictive Analysis: A synthetic column (Annual Purchase Amount) is included for testing linear regression, allowing users to explore predictive analysis features.
Getting Started with Docker
You can run PipeDreams using Docker to avoid setting up dependencies locally. The pre-built Docker image is available on DockerHub.
Pulling the Docker Image
Pull the latest Docker image from DockerHub:
docker pull anuclei/pipedreams:latest
Running the Docker Container
Run the application with Docker, exposing it on port 8501:
docker run -p 8501:8501 anuclei/pipedreams:latest
Once the container is running, open your browser and go to http://localhost:8501 to access the application.
Kubernetes Deployment
PipeDreams can also be deployed on a Kubernetes cluster. This deployment scenario uses Minikube for local Kubernetes clusters and includes configurations for high availability and autoscaling.
For detailed instructions and YAML configurations, refer to the Kubernetes Deployment Guide in the k8s directory.
Manual Installation
If you prefer not to use Docker, you can set up the app manually.
Prerequisites
  • Python (version 3.6 or higher)
Installation
  1. Clone the repository:
    git clone https://github.com/markjacksonfishing/pipedreams.git
cd pipedreams
  2. Set up a virtual environment:
    • MacOS/Linux:

      python3 -m venv venv
source venv/bin/activate
    • Windows:

      python -m venv venv
.\venv\Scripts\activate
  3. Install dependencies:
    pip install -r requirements.txt
  4. Run the application:
    • MacOS/Linux:

      source venv/bin/activate
streamlit run app.py
    • Windows:

      .\venv\Scripts\activate
streamlit run app.py
    The application will open in your default web browser at http://localhost:8501 and will look like this:
PipeDreams Browser
  5. Deactivate the virtual environment (when finished):
    deactivate
How to Use
  1. Start the application by following the setup steps above (or run it via Docker).
  2. Upload a CSV file using the file uploader in the app, or view the default dataset if no file is uploaded.
  3. Explore the data with built-in ETL transformations and interactive visualizations.
  4. Perform clustering analysis and predictive analysis on available data.
Advanced Insights: Clustering and Predictive Analysis
  • Clustering Analysis: Select features for clustering, and the app will automatically group data into clusters using KMeans. This can reveal natural groupings in the data, such as customer segments.
  • Predictive Analysis: Select features and a target variable (e.g., the synthetic Annual Purchase Amount) for linear regression. The app will generate a prediction model, display a mean squared error metric, and show an interactive scatter plot comparing actual vs. predicted values.
Default Dataset: customers-100000.csv
The default dataset, customers-100000.csv, is located in the data/ directory. If no CSV file is uploaded, this dataset will automatically load, allowing users to test the ETL transformations, visualizations, clustering, and predictive analysis features without needing their own data file.
About
A play on pipelines, with a focus on making data accessible and insightful.
www.anuclei.com
Topics
backenddata-engineeringdata-processingdata-visualizationdeploymentetlfrontendmachine-learningpythonstreamlit
Resources
Readme
MIT license
Activity
Stars
0 stars
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - markjacksonfishing/pipedreams: A play on pipelines, with a focus on making data accessible and insightful. · GitHub
Skip to content
BranchesTags
Repository files navigation
PipeDreams - CSV Data Explorer
PipeDreams Header
PipeDreams is a data exploration and visualization tool designed to be simple, flexible, and powerful. Upload any CSV file, perform basic ETL transformations, visualize your data, and gain insights with built-in machine learning—all from a user-friendly Streamlit interface. If no file is uploaded, a default dataset (customers-100000.csv) is used for demonstration.
Features
  • CSV Upload: Upload any CSV file for immediate analysis and visualization.
  • Default Dataset: If no file is uploaded, the app loads a sample dataset (customers-100000.csv) located in the data/ directory.
  • ETL Transformations: Clean and transform data, remove missing values, and auto-convert data types.
  • Data Visualization: Interactive charts (scatter, bar, line, histogram, and box plots) to gain insights from your data.
  • Clustering Analysis: Use KMeans clustering to identify natural groupings within the data, helping to segment and classify.
  • Predictive Analysis: A synthetic column (Annual Purchase Amount) is included for testing linear regression, allowing users to explore predictive analysis features.
Getting Started with Docker
You can run PipeDreams using Docker to avoid setting up dependencies locally. The pre-built Docker image is available on DockerHub.
Pulling the Docker Image
Pull the latest Docker image from DockerHub:
docker pull anuclei/pipedreams:latest
Running the Docker Container
Run the application with Docker, exposing it on port 8501:
docker run -p 8501:8501 anuclei/pipedreams:latest
Once the container is running, open your browser and go to http://localhost:8501 to access the application.
Kubernetes Deployment
PipeDreams can also be deployed on a Kubernetes cluster. This deployment scenario uses Minikube for local Kubernetes clusters and includes configurations for high availability and autoscaling.
For detailed instructions and YAML configurations, refer to the Kubernetes Deployment Guide in the k8s directory.
Manual Installation
If you prefer not to use Docker, you can set up the app manually.
Prerequisites
  • Python (version 3.6 or higher)
Installation
  1. Clone the repository:
    git clone https://github.com/markjacksonfishing/pipedreams.git
cd pipedreams
  2. Set up a virtual environment:
    • MacOS/Linux:

      python3 -m venv venv
source venv/bin/activate
    • Windows:

      python -m venv venv
.\venv\Scripts\activate
  3. Install dependencies:
    pip install -r requirements.txt
  4. Run the application:
    • MacOS/Linux:

      source venv/bin/activate
streamlit run app.py
    • Windows:

      .\venv\Scripts\activate
streamlit run app.py
    The application will open in your default web browser at http://localhost:8501 and will look like this:
PipeDreams Browser
  5. Deactivate the virtual environment (when finished):
    deactivate
How to Use
  1. Start the application by following the setup steps above (or run it via Docker).
  2. Upload a CSV file using the file uploader in the app, or view the default dataset if no file is uploaded.
  3. Explore the data with built-in ETL transformations and interactive visualizations.
  4. Perform clustering analysis and predictive analysis on available data.
Advanced Insights: Clustering and Predictive Analysis
  • Clustering Analysis: Select features for clustering, and the app will automatically group data into clusters using KMeans. This can reveal natural groupings in the data, such as customer segments.
  • Predictive Analysis: Select features and a target variable (e.g., the synthetic Annual Purchase Amount) for linear regression. The app will generate a prediction model, display a mean squared error metric, and show an interactive scatter plot comparing actual vs. predicted values.
Default Dataset: customers-100000.csv
The default dataset, customers-100000.csv, is located in the data/ directory. If no CSV file is uploaded, this dataset will automatically load, allowing users to test the ETL transformations, visualizations, clustering, and predictive analysis features without needing their own data file.
About
A play on pipelines, with a focus on making data accessible and insightful.
www.anuclei.com
Topics
backenddata-engineeringdata-processingdata-visualizationdeploymentetlfrontendmachine-learningpythonstreamlit
Resources
Readme
MIT license
Activity
Stars
0 stars
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
GitHub - markjacksonfishing/pipedreams: A play on pipelines, with a focus on making data accessible and insightful. · GitHub
Skip to content
BranchesTags
Repository files navigation
PipeDreams - CSV Data Explorer
PipeDreams Header
PipeDreams is a data exploration and visualization tool designed to be simple, flexible, and powerful. Upload any CSV file, perform basic ETL transformations, visualize your data, and gain insights with built-in machine learning—all from a user-friendly Streamlit interface. If no file is uploaded, a default dataset (customers-100000.csv) is used for demonstration.
Features
  • CSV Upload: Upload any CSV file for immediate analysis and visualization.
  • Default Dataset: If no file is uploaded, the app loads a sample dataset (customers-100000.csv) located in the data/ directory.
  • ETL Transformations: Clean and transform data, remove missing values, and auto-convert data types.
  • Data Visualization: Interactive charts (scatter, bar, line, histogram, and box plots) to gain insights from your data.
  • Clustering Analysis: Use KMeans clustering to identify natural groupings within the data, helping to segment and classify.
  • Predictive Analysis: A synthetic column (Annual Purchase Amount) is included for testing linear regression, allowing users to explore predictive analysis features.
Getting Started with Docker
You can run PipeDreams using Docker to avoid setting up dependencies locally. The pre-built Docker image is available on DockerHub.
Pulling the Docker Image
Pull the latest Docker image from DockerHub:
docker pull anuclei/pipedreams:latest
Running the Docker Container
Run the application with Docker, exposing it on port 8501:
docker run -p 8501:8501 anuclei/pipedreams:latest
Once the container is running, open your browser and go to http://localhost:8501 to access the application.
Kubernetes Deployment
PipeDreams can also be deployed on a Kubernetes cluster. This deployment scenario uses Minikube for local Kubernetes clusters and includes configurations for high availability and autoscaling.
For detailed instructions and YAML configurations, refer to the Kubernetes Deployment Guide in the k8s directory.
Manual Installation
If you prefer not to use Docker, you can set up the app manually.
Prerequisites
  • Python (version 3.6 or higher)
Installation
  1. Clone the repository:
    git clone https://github.com/markjacksonfishing/pipedreams.git
cd pipedreams
  2. Set up a virtual environment:
    • MacOS/Linux:

      python3 -m venv venv
source venv/bin/activate
    • Windows:

      python -m venv venv
.\venv\Scripts\activate
  3. Install dependencies:
    pip install -r requirements.txt
  4. Run the application:
    • MacOS/Linux:

      source venv/bin/activate
streamlit run app.py
    • Windows:

      .\venv\Scripts\activate
streamlit run app.py
    The application will open in your default web browser at http://localhost:8501 and will look like this:
PipeDreams Browser
  5. Deactivate the virtual environment (when finished):
    deactivate
How to Use
  1. Start the application by following the setup steps above (or run it via Docker).
  2. Upload a CSV file using the file uploader in the app, or view the default dataset if no file is uploaded.
  3. Explore the data with built-in ETL transformations and interactive visualizations.
  4. Perform clustering analysis and predictive analysis on available data.
Advanced Insights: Clustering and Predictive Analysis
  • Clustering Analysis: Select features for clustering, and the app will automatically group data into clusters using KMeans. This can reveal natural groupings in the data, such as customer segments.
  • Predictive Analysis: Select features and a target variable (e.g., the synthetic Annual Purchase Amount) for linear regression. The app will generate a prediction model, display a mean squared error metric, and show an interactive scatter plot comparing actual vs. predicted values.
Default Dataset: customers-100000.csv
The default dataset, customers-100000.csv, is located in the data/ directory. If no CSV file is uploaded, this dataset will automatically load, allowing users to test the ETL transformations, visualizations, clustering, and predictive analysis features without needing their own data file.
About
A play on pipelines, with a focus on making data accessible and insightful.
www.anuclei.com
Topics
backenddata-engineeringdata-processingdata-visualizationdeploymentetlfrontendmachine-learningpythonstreamlit
Resources
Readme
MIT license
Activity
Stars
0 stars
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - markjacksonfishing/pipedreams: A play on pipelines, with a focus on making data accessible and insightful. · GitHub
Skip to content
BranchesTags
Repository files navigation
PipeDreams - CSV Data Explorer
PipeDreams Header
PipeDreams is a data exploration and visualization tool designed to be simple, flexible, and powerful. Upload any CSV file, perform basic ETL transformations, visualize your data, and gain insights with built-in machine learning—all from a user-friendly Streamlit interface. If no file is uploaded, a default dataset (customers-100000.csv) is used for demonstration.
Features
  • CSV Upload: Upload any CSV file for immediate analysis and visualization.
  • Default Dataset: If no file is uploaded, the app loads a sample dataset (customers-100000.csv) located in the data/ directory.
  • ETL Transformations: Clean and transform data, remove missing values, and auto-convert data types.
  • Data Visualization: Interactive charts (scatter, bar, line, histogram, and box plots) to gain insights from your data.
  • Clustering Analysis: Use KMeans clustering to identify natural groupings within the data, helping to segment and classify.
  • Predictive Analysis: A synthetic column (Annual Purchase Amount) is included for testing linear regression, allowing users to explore predictive analysis features.
Getting Started with Docker
You can run PipeDreams using Docker to avoid setting up dependencies locally. The pre-built Docker image is available on DockerHub.
Pulling the Docker Image
Pull the latest Docker image from DockerHub:
docker pull anuclei/pipedreams:latest
Running the Docker Container
Run the application with Docker, exposing it on port 8501:
docker run -p 8501:8501 anuclei/pipedreams:latest
Once the container is running, open your browser and go to http://localhost:8501 to access the application.
Kubernetes Deployment
PipeDreams can also be deployed on a Kubernetes cluster. This deployment scenario uses Minikube for local Kubernetes clusters and includes configurations for high availability and autoscaling.
For detailed instructions and YAML configurations, refer to the Kubernetes Deployment Guide in the k8s directory.
Manual Installation
If you prefer not to use Docker, you can set up the app manually.
Prerequisites
  • Python (version 3.6 or higher)
Installation
  1. Clone the repository:
    git clone https://github.com/markjacksonfishing/pipedreams.git
cd pipedreams
  2. Set up a virtual environment:
    • MacOS/Linux:

      python3 -m venv venv
source venv/bin/activate
    • Windows:

      python -m venv venv
.\venv\Scripts\activate
  3. Install dependencies:
    pip install -r requirements.txt
  4. Run the application:
    • MacOS/Linux:

      source venv/bin/activate
streamlit run app.py
    • Windows:

      .\venv\Scripts\activate
streamlit run app.py
    The application will open in your default web browser at http://localhost:8501 and will look like this:
PipeDreams Browser
  5. Deactivate the virtual environment (when finished):
    deactivate
How to Use
  1. Start the application by following the setup steps above (or run it via Docker).
  2. Upload a CSV file using the file uploader in the app, or view the default dataset if no file is uploaded.
  3. Explore the data with built-in ETL transformations and interactive visualizations.
  4. Perform clustering analysis and predictive analysis on available data.
Advanced Insights: Clustering and Predictive Analysis
  • Clustering Analysis: Select features for clustering, and the app will automatically group data into clusters using KMeans. This can reveal natural groupings in the data, such as customer segments.
  • Predictive Analysis: Select features and a target variable (e.g., the synthetic Annual Purchase Amount) for linear regression. The app will generate a prediction model, display a mean squared error metric, and show an interactive scatter plot comparing actual vs. predicted values.
Default Dataset: customers-100000.csv
The default dataset, customers-100000.csv, is located in the data/ directory. If no CSV file is uploaded, this dataset will automatically load, allowing users to test the ETL transformations, visualizations, clustering, and predictive analysis features without needing their own data file.
About
A play on pipelines, with a focus on making data accessible and insightful.
www.anuclei.com
Topics
backenddata-engineeringdata-processingdata-visualizationdeploymentetlfrontendmachine-learningpythonstreamlit
Resources
Readme
MIT license
Activity
Stars
0 stars
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - markjacksonfishing/pipedreams: A play on pipelines, with a focus on making data accessible and insightful. · GitHub
Skip to content
BranchesTags
Repository files navigation
PipeDreams - CSV Data Explorer
PipeDreams Header
PipeDreams is a data exploration and visualization tool designed to be simple, flexible, and powerful. Upload any CSV file, perform basic ETL transformations, visualize your data, and gain insights with built-in machine learning—all from a user-friendly Streamlit interface. If no file is uploaded, a default dataset (customers-100000.csv) is used for demonstration.
Features
  • CSV Upload: Upload any CSV file for immediate analysis and visualization.
  • Default Dataset: If no file is uploaded, the app loads a sample dataset (customers-100000.csv) located in the data/ directory.
  • ETL Transformations: Clean and transform data, remove missing values, and auto-convert data types.
  • Data Visualization: Interactive charts (scatter, bar, line, histogram, and box plots) to gain insights from your data.
  • Clustering Analysis: Use KMeans clustering to identify natural groupings within the data, helping to segment and classify.
  • Predictive Analysis: A synthetic column (Annual Purchase Amount) is included for testing linear regression, allowing users to explore predictive analysis features.
Getting Started with Docker
You can run PipeDreams using Docker to avoid setting up dependencies locally. The pre-built Docker image is available on DockerHub.
Pulling the Docker Image
Pull the latest Docker image from DockerHub:
docker pull anuclei/pipedreams:latest
Running the Docker Container
Run the application with Docker, exposing it on port 8501:
docker run -p 8501:8501 anuclei/pipedreams:latest
Once the container is running, open your browser and go to http://localhost:8501 to access the application.
Kubernetes Deployment
PipeDreams can also be deployed on a Kubernetes cluster. This deployment scenario uses Minikube for local Kubernetes clusters and includes configurations for high availability and autoscaling.
For detailed instructions and YAML configurations, refer to the Kubernetes Deployment Guide in the k8s directory.
Manual Installation
If you prefer not to use Docker, you can set up the app manually.
Prerequisites
  • Python (version 3.6 or higher)
Installation
  1. Clone the repository:
    git clone https://github.com/markjacksonfishing/pipedreams.git
cd pipedreams
  2. Set up a virtual environment:
    • MacOS/Linux:

      python3 -m venv venv
source venv/bin/activate
    • Windows:

      python -m venv venv
.\venv\Scripts\activate
  3. Install dependencies:
    pip install -r requirements.txt
  4. Run the application:
    • MacOS/Linux:

      source venv/bin/activate
streamlit run app.py
    • Windows:

      .\venv\Scripts\activate
streamlit run app.py
    The application will open in your default web browser at http://localhost:8501 and will look like this:
PipeDreams Browser
  5. Deactivate the virtual environment (when finished):
    deactivate
How to Use
  1. Start the application by following the setup steps above (or run it via Docker).
  2. Upload a CSV file using the file uploader in the app, or view the default dataset if no file is uploaded.
  3. Explore the data with built-in ETL transformations and interactive visualizations.
  4. Perform clustering analysis and predictive analysis on available data.
Advanced Insights: Clustering and Predictive Analysis
  • Clustering Analysis: Select features for clustering, and the app will automatically group data into clusters using KMeans. This can reveal natural groupings in the data, such as customer segments.
  • Predictive Analysis: Select features and a target variable (e.g., the synthetic Annual Purchase Amount) for linear regression. The app will generate a prediction model, display a mean squared error metric, and show an interactive scatter plot comparing actual vs. predicted values.
Default Dataset: customers-100000.csv
The default dataset, customers-100000.csv, is located in the data/ directory. If no CSV file is uploaded, this dataset will automatically load, allowing users to test the ETL transformations, visualizations, clustering, and predictive analysis features without needing their own data file.
About
A play on pipelines, with a focus on making data accessible and insightful.
www.anuclei.com
Topics
backenddata-engineeringdata-processingdata-visualizationdeploymentetlfrontendmachine-learningpythonstreamlit
Resources
Readme
MIT license
Activity
Stars
0 stars
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
GitHub - markjacksonfishing/pipedreams: A play on pipelines, with a focus on making data accessible and insightful. · GitHub
Skip to content
BranchesTags
Repository files navigation
PipeDreams - CSV Data Explorer
PipeDreams Header
PipeDreams is a data exploration and visualization tool designed to be simple, flexible, and powerful. Upload any CSV file, perform basic ETL transformations, visualize your data, and gain insights with built-in machine learning—all from a user-friendly Streamlit interface. If no file is uploaded, a default dataset (customers-100000.csv) is used for demonstration.
Features
  • CSV Upload: Upload any CSV file for immediate analysis and visualization.
  • Default Dataset: If no file is uploaded, the app loads a sample dataset (customers-100000.csv) located in the data/ directory.
  • ETL Transformations: Clean and transform data, remove missing values, and auto-convert data types.
  • Data Visualization: Interactive charts (scatter, bar, line, histogram, and box plots) to gain insights from your data.
  • Clustering Analysis: Use KMeans clustering to identify natural groupings within the data, helping to segment and classify.
  • Predictive Analysis: A synthetic column (Annual Purchase Amount) is included for testing linear regression, allowing users to explore predictive analysis features.
Getting Started with Docker
You can run PipeDreams using Docker to avoid setting up dependencies locally. The pre-built Docker image is available on DockerHub.
Pulling the Docker Image
Pull the latest Docker image from DockerHub:
docker pull anuclei/pipedreams:latest
Running the Docker Container
Run the application with Docker, exposing it on port 8501:
docker run -p 8501:8501 anuclei/pipedreams:latest
Once the container is running, open your browser and go to http://localhost:8501 to access the application.
Kubernetes Deployment
PipeDreams can also be deployed on a Kubernetes cluster. This deployment scenario uses Minikube for local Kubernetes clusters and includes configurations for high availability and autoscaling.
For detailed instructions and YAML configurations, refer to the Kubernetes Deployment Guide in the k8s directory.
Manual Installation
If you prefer not to use Docker, you can set up the app manually.
Prerequisites
  • Python (version 3.6 or higher)
Installation
  1. Clone the repository:
    git clone https://github.com/markjacksonfishing/pipedreams.git
cd pipedreams
  2. Set up a virtual environment:
    • MacOS/Linux:

      python3 -m venv venv
source venv/bin/activate
    • Windows:

      python -m venv venv
.\venv\Scripts\activate
  3. Install dependencies:
    pip install -r requirements.txt
  4. Run the application:
    • MacOS/Linux:

      source venv/bin/activate
streamlit run app.py
    • Windows:

      .\venv\Scripts\activate
streamlit run app.py
    The application will open in your default web browser at http://localhost:8501 and will look like this:
PipeDreams Browser
  5. Deactivate the virtual environment (when finished):
    deactivate
How to Use
  1. Start the application by following the setup steps above (or run it via Docker).
  2. Upload a CSV file using the file uploader in the app, or view the default dataset if no file is uploaded.
  3. Explore the data with built-in ETL transformations and interactive visualizations.
  4. Perform clustering analysis and predictive analysis on available data.
Advanced Insights: Clustering and Predictive Analysis
  • Clustering Analysis: Select features for clustering, and the app will automatically group data into clusters using KMeans. This can reveal natural groupings in the data, such as customer segments.
  • Predictive Analysis: Select features and a target variable (e.g., the synthetic Annual Purchase Amount) for linear regression. The app will generate a prediction model, display a mean squared error metric, and show an interactive scatter plot comparing actual vs. predicted values.
Default Dataset: customers-100000.csv
The default dataset, customers-100000.csv, is located in the data/ directory. If no CSV file is uploaded, this dataset will automatically load, allowing users to test the ETL transformations, visualizations, clustering, and predictive analysis features without needing their own data file.
About
A play on pipelines, with a focus on making data accessible and insightful.
www.anuclei.com
Topics
backenddata-engineeringdata-processingdata-visualizationdeploymentetlfrontendmachine-learningpythonstreamlit
Resources
Readme
MIT license
Activity
Stars
0 stars
Watchers
1 watching
Forks
0 forks
Report repository
Releases
Packages
Used by
Contributors
Languages