Skip to content
BranchesTags

Latest commit

History

19 Commits

Folders and files

NameName
Last commit message
Last commit date
lab_instruction
lab_instruction
src
src
README.md
README.md

Repository files navigation

Mapreduce

This repo contains mapreduce implementation in Go.

This is first lab in MIT's course on distributed systems 6.824. Please see the lab instruction here. Note that I didn't submit the lab, since I wasn't enrolled.

Overview

We have a master and multiple workers.

  1. User starts a master by specifying number of reducers (nR) and a list of files to run mapReduce on. User also starts multiple workers with custom mapper/reducer funtions
  2. Master stores the list of files the users provided and other state in a master Data structure.
  3. When workers are started, they ping master for task every 5 sec (its synchronous, so worker finishes the task at hand before pinging the master again )
  4. During map phase:
    • master gives worker one file to run map
    • worker runs mapper on each file. for each map task (i-th file), the worker produces bunch of key, values. Each key,value is stored as a json entry in a file called mr-{{i}}-{{r}}.txt where i is the file index and r = hash(key) mod nR. So each map tasks produces at most rn files. So overall after all map tasks are done there will be numFiles * nR files.
  5. During Reduce phase:
    • there will be nR reduce tasks. For each r-th task, master gives worker a reduce task with ID = r and a list of all files which have key, values such that r = hash(key) mod nR namely the files mr-*-{{r}}.txt.
    • the output of each reduce task will be in a file name mr-out-r

Handling worker crash

The master will start a goroutine to monitor state of tasks. It checks if any tasks are not marked DONE within 10 secs. If so, they are marked FAILED and eligible for reassignment. One of the map reduce apps is a crash.go which crashes or runs slow with certain probabilities. The test file src/main/test-mr.sh produces scenario to run the crash scenario.

How to run

  • Start master with a list of files as argument
cd src/main
go run mrmaster.go pg-being_ernest.txt pg-grimm.txt
  • Start some workers with a mapreduce plugin (eg wordcounter)
cd src/main
# compile plugin
go build -buildmode=plugin ../mrapps/wc.go
# run worker
go run mrworker.go wc.so

Test

cd src/main
sh test-mr.sh

TODO

This project is not ready for distributed envronment. All the mappers produce the intermediate files in the same directory which reducers can access. In distributed environment, the reducers will either have to connect to mapper via RPC to access the files or the mappers will have to store files in a distributed file system (eg HDFS)

About

Implementation of MapReduce in Go

Resources

Readme
Activity

Stars

8 stars

Watchers

1 watching

Forks

3 forks
Report repository

Releases

Packages

Used by

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
GitHub - vvksh/mapreduce: Implementation of MapReduce in Go · GitHub
Skip to content
BranchesTags
Latest commit
History
19 Commits
Folders and files
NameName
Last commit message
Last commit date
lab_instruction
lab_instruction
src
src
README.md
README.md
Repository files navigation
Mapreduce
This repo contains mapreduce implementation in Go.
This is first lab in MIT's course on distributed systems 6.824. Please see the lab instruction here. Note that I didn't submit the lab, since I wasn't enrolled.
Overview
We have a master and multiple workers.
  1. User starts a master by specifying number of reducers (nR) and a list of files to run mapReduce on. User also starts multiple workers with custom mapper/reducer funtions
  2. Master stores the list of files the users provided and other state in a master Data structure.
  3. When workers are started, they ping master for task every 5 sec (its synchronous, so worker finishes the task at hand before pinging the master again )
  4. During map phase:

    • master gives worker one file to run map
    • worker runs mapper on each file. for each map task (i-th file), the worker produces bunch of key, values. Each key,value is stored as a json entry in a file called mr-{{i}}-{{r}}.txt where i is the file index and r = hash(key) mod nR. So each map tasks produces at most rn files. So overall after all map tasks are done there will be numFiles * nR files.
  5. During Reduce phase:

    • there will be nR reduce tasks. For each r-th task, master gives worker a reduce task with ID = r and a list of all files which have key, values such that r = hash(key) mod nR namely the files
mr-*-{{r}}.txt.
    • the output of each reduce task will be in a file name mr-out-r
Handling worker crash
The master will start a goroutine to monitor state of tasks. It checks if any tasks are not marked DONE within 10 secs. If so, they are marked FAILED and eligible for reassignment.
One of the map reduce apps is a crash.go which crashes or runs slow with certain probabilities. The test file src/main/test-mr.sh produces scenario to run the crash scenario.
How to run
  • Start master with a list of files as argument
cd src/main
go run mrmaster.go pg-being_ernest.txt pg-grimm.txt
  • Start some workers with a mapreduce plugin (eg wordcounter)
cd src/main
# compile plugin
go build -buildmode=plugin ../mrapps/wc.go
# run worker
go run mrworker.go wc.so

Test
cd src/main
sh test-mr.sh

TODO
This project is not ready for distributed envronment. All the mappers produce the intermediate files in the same directory which reducers can access.
In distributed environment, the reducers will either have to connect to mapper via RPC to access the files or the mappers will have to store files in
a distributed file system (eg HDFS)
About
Implementation of MapReduce in Go
Resources
Readme
Activity
Stars
8 stars
Watchers
1 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - vvksh/mapreduce: Implementation of MapReduce in Go · GitHub
Skip to content
BranchesTags
Latest commit
History
19 Commits
Folders and files
NameName
Last commit message
Last commit date
lab_instruction
lab_instruction
src
src
README.md
README.md
Repository files navigation
Mapreduce
This repo contains mapreduce implementation in Go.
This is first lab in MIT's course on distributed systems 6.824. Please see the lab instruction here. Note that I didn't submit the lab, since I wasn't enrolled.
Overview
We have a master and multiple workers.
  1. User starts a master by specifying number of reducers (nR) and a list of files to run mapReduce on. User also starts multiple workers with custom mapper/reducer funtions
  2. Master stores the list of files the users provided and other state in a master Data structure.
  3. When workers are started, they ping master for task every 5 sec (its synchronous, so worker finishes the task at hand before pinging the master again )
  4. During map phase:

    • master gives worker one file to run map
    • worker runs mapper on each file. for each map task (i-th file), the worker produces bunch of key, values. Each key,value is stored as a json entry in a file called mr-{{i}}-{{r}}.txt where i is the file index and r = hash(key) mod nR. So each map tasks produces at most rn files. So overall after all map tasks are done there will be numFiles * nR files.
  5. During Reduce phase:

    • there will be nR reduce tasks. For each r-th task, master gives worker a reduce task with ID = r and a list of all files which have key, values such that r = hash(key) mod nR namely the files
mr-*-{{r}}.txt.
    • the output of each reduce task will be in a file name mr-out-r
Handling worker crash
The master will start a goroutine to monitor state of tasks. It checks if any tasks are not marked DONE within 10 secs. If so, they are marked FAILED and eligible for reassignment.
One of the map reduce apps is a crash.go which crashes or runs slow with certain probabilities. The test file src/main/test-mr.sh produces scenario to run the crash scenario.
How to run
  • Start master with a list of files as argument
cd src/main
go run mrmaster.go pg-being_ernest.txt pg-grimm.txt
  • Start some workers with a mapreduce plugin (eg wordcounter)
cd src/main
# compile plugin
go build -buildmode=plugin ../mrapps/wc.go
# run worker
go run mrworker.go wc.so

Test
cd src/main
sh test-mr.sh

TODO
This project is not ready for distributed envronment. All the mappers produce the intermediate files in the same directory which reducers can access.
In distributed environment, the reducers will either have to connect to mapper via RPC to access the files or the mappers will have to store files in
a distributed file system (eg HDFS)
About
Implementation of MapReduce in Go
Resources
Readme
Activity
Stars
8 stars
Watchers
1 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - vvksh/mapreduce: Implementation of MapReduce in Go · GitHub
Skip to content
BranchesTags
Latest commit
History
19 Commits
Folders and files
NameName
Last commit message
Last commit date
lab_instruction
lab_instruction
src
src
README.md
README.md
Repository files navigation
Mapreduce
This repo contains mapreduce implementation in Go.
This is first lab in MIT's course on distributed systems 6.824. Please see the lab instruction here. Note that I didn't submit the lab, since I wasn't enrolled.
Overview
We have a master and multiple workers.
  1. User starts a master by specifying number of reducers (nR) and a list of files to run mapReduce on. User also starts multiple workers with custom mapper/reducer funtions
  2. Master stores the list of files the users provided and other state in a master Data structure.
  3. When workers are started, they ping master for task every 5 sec (its synchronous, so worker finishes the task at hand before pinging the master again )
  4. During map phase:

    • master gives worker one file to run map
    • worker runs mapper on each file. for each map task (i-th file), the worker produces bunch of key, values. Each key,value is stored as a json entry in a file called mr-{{i}}-{{r}}.txt where i is the file index and r = hash(key) mod nR. So each map tasks produces at most rn files. So overall after all map tasks are done there will be numFiles * nR files.
  5. During Reduce phase:

    • there will be nR reduce tasks. For each r-th task, master gives worker a reduce task with ID = r and a list of all files which have key, values such that r = hash(key) mod nR namely the files
mr-*-{{r}}.txt.
    • the output of each reduce task will be in a file name mr-out-r
Handling worker crash
The master will start a goroutine to monitor state of tasks. It checks if any tasks are not marked DONE within 10 secs. If so, they are marked FAILED and eligible for reassignment.
One of the map reduce apps is a crash.go which crashes or runs slow with certain probabilities. The test file src/main/test-mr.sh produces scenario to run the crash scenario.
How to run
  • Start master with a list of files as argument
cd src/main
go run mrmaster.go pg-being_ernest.txt pg-grimm.txt
  • Start some workers with a mapreduce plugin (eg wordcounter)
cd src/main
# compile plugin
go build -buildmode=plugin ../mrapps/wc.go
# run worker
go run mrworker.go wc.so

Test
cd src/main
sh test-mr.sh

TODO
This project is not ready for distributed envronment. All the mappers produce the intermediate files in the same directory which reducers can access.
In distributed environment, the reducers will either have to connect to mapper via RPC to access the files or the mappers will have to store files in
a distributed file system (eg HDFS)
About
Implementation of MapReduce in Go
Resources
Readme
Activity
Stars
8 stars
Watchers
1 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '
GitHub - vvksh/mapreduce: Implementation of MapReduce in Go · GitHub
Skip to content
BranchesTags
Latest commit
History
19 Commits
Folders and files
NameName
Last commit message
Last commit date
lab_instruction
lab_instruction
src
src
README.md
README.md
Repository files navigation
Mapreduce
This repo contains mapreduce implementation in Go.
This is first lab in MIT's course on distributed systems 6.824. Please see the lab instruction here. Note that I didn't submit the lab, since I wasn't enrolled.
Overview
We have a master and multiple workers.
  1. User starts a master by specifying number of reducers (nR) and a list of files to run mapReduce on. User also starts multiple workers with custom mapper/reducer funtions
  2. Master stores the list of files the users provided and other state in a master Data structure.
  3. When workers are started, they ping master for task every 5 sec (its synchronous, so worker finishes the task at hand before pinging the master again )
  4. During map phase:

    • master gives worker one file to run map
    • worker runs mapper on each file. for each map task (i-th file), the worker produces bunch of key, values. Each key,value is stored as a json entry in a file called mr-{{i}}-{{r}}.txt where i is the file index and r = hash(key) mod nR. So each map tasks produces at most rn files. So overall after all map tasks are done there will be numFiles * nR files.
  5. During Reduce phase:

    • there will be nR reduce tasks. For each r-th task, master gives worker a reduce task with ID = r and a list of all files which have key, values such that r = hash(key) mod nR namely the files
mr-*-{{r}}.txt.
    • the output of each reduce task will be in a file name mr-out-r
Handling worker crash
The master will start a goroutine to monitor state of tasks. It checks if any tasks are not marked DONE within 10 secs. If so, they are marked FAILED and eligible for reassignment.
One of the map reduce apps is a crash.go which crashes or runs slow with certain probabilities. The test file src/main/test-mr.sh produces scenario to run the crash scenario.
How to run
  • Start master with a list of files as argument
cd src/main
go run mrmaster.go pg-being_ernest.txt pg-grimm.txt
  • Start some workers with a mapreduce plugin (eg wordcounter)
cd src/main
# compile plugin
go build -buildmode=plugin ../mrapps/wc.go
# run worker
go run mrworker.go wc.so

Test
cd src/main
sh test-mr.sh

TODO
This project is not ready for distributed envronment. All the mappers produce the intermediate files in the same directory which reducers can access.
In distributed environment, the reducers will either have to connect to mapper via RPC to access the files or the mappers will have to store files in
a distributed file system (eg HDFS)
About
Implementation of MapReduce in Go
Resources
Readme
Activity
Stars
8 stars
Watchers
1 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - vvksh/mapreduce: Implementation of MapReduce in Go · GitHub
Skip to content
BranchesTags
Latest commit
History
19 Commits
Folders and files
NameName
Last commit message
Last commit date
lab_instruction
lab_instruction
src
src
README.md
README.md
Repository files navigation
Mapreduce
This repo contains mapreduce implementation in Go.
This is first lab in MIT's course on distributed systems 6.824. Please see the lab instruction here. Note that I didn't submit the lab, since I wasn't enrolled.
Overview
We have a master and multiple workers.
  1. User starts a master by specifying number of reducers (nR) and a list of files to run mapReduce on. User also starts multiple workers with custom mapper/reducer funtions
  2. Master stores the list of files the users provided and other state in a master Data structure.
  3. When workers are started, they ping master for task every 5 sec (its synchronous, so worker finishes the task at hand before pinging the master again )
  4. During map phase:

    • master gives worker one file to run map
    • worker runs mapper on each file. for each map task (i-th file), the worker produces bunch of key, values. Each key,value is stored as a json entry in a file called mr-{{i}}-{{r}}.txt where i is the file index and r = hash(key) mod nR. So each map tasks produces at most rn files. So overall after all map tasks are done there will be numFiles * nR files.
  5. During Reduce phase:

    • there will be nR reduce tasks. For each r-th task, master gives worker a reduce task with ID = r and a list of all files which have key, values such that r = hash(key) mod nR namely the files
mr-*-{{r}}.txt.
    • the output of each reduce task will be in a file name mr-out-r
Handling worker crash
The master will start a goroutine to monitor state of tasks. It checks if any tasks are not marked DONE within 10 secs. If so, they are marked FAILED and eligible for reassignment.
One of the map reduce apps is a crash.go which crashes or runs slow with certain probabilities. The test file src/main/test-mr.sh produces scenario to run the crash scenario.
How to run
  • Start master with a list of files as argument
cd src/main
go run mrmaster.go pg-being_ernest.txt pg-grimm.txt
  • Start some workers with a mapreduce plugin (eg wordcounter)
cd src/main
# compile plugin
go build -buildmode=plugin ../mrapps/wc.go
# run worker
go run mrworker.go wc.so

Test
cd src/main
sh test-mr.sh

TODO
This project is not ready for distributed envronment. All the mappers produce the intermediate files in the same directory which reducers can access.
In distributed environment, the reducers will either have to connect to mapper via RPC to access the files or the mappers will have to store files in
a distributed file system (eg HDFS)
About
Implementation of MapReduce in Go
Resources
Readme
Activity
Stars
8 stars
Watchers
1 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '
GitHub - vvksh/mapreduce: Implementation of MapReduce in Go · GitHub
Skip to content
BranchesTags
Latest commit
History
19 Commits
Folders and files
NameName
Last commit message
Last commit date
lab_instruction
lab_instruction
src
src
README.md
README.md
Repository files navigation
Mapreduce
This repo contains mapreduce implementation in Go.
This is first lab in MIT's course on distributed systems 6.824. Please see the lab instruction here. Note that I didn't submit the lab, since I wasn't enrolled.
Overview
We have a master and multiple workers.
  1. User starts a master by specifying number of reducers (nR) and a list of files to run mapReduce on. User also starts multiple workers with custom mapper/reducer funtions
  2. Master stores the list of files the users provided and other state in a master Data structure.
  3. When workers are started, they ping master for task every 5 sec (its synchronous, so worker finishes the task at hand before pinging the master again )
  4. During map phase:

    • master gives worker one file to run map
    • worker runs mapper on each file. for each map task (i-th file), the worker produces bunch of key, values. Each key,value is stored as a json entry in a file called mr-{{i}}-{{r}}.txt where i is the file index and r = hash(key) mod nR. So each map tasks produces at most rn files. So overall after all map tasks are done there will be numFiles * nR files.
  5. During Reduce phase:

    • there will be nR reduce tasks. For each r-th task, master gives worker a reduce task with ID = r and a list of all files which have key, values such that r = hash(key) mod nR namely the files
mr-*-{{r}}.txt.
    • the output of each reduce task will be in a file name mr-out-r
Handling worker crash
The master will start a goroutine to monitor state of tasks. It checks if any tasks are not marked DONE within 10 secs. If so, they are marked FAILED and eligible for reassignment.
One of the map reduce apps is a crash.go which crashes or runs slow with certain probabilities. The test file src/main/test-mr.sh produces scenario to run the crash scenario.
How to run
  • Start master with a list of files as argument
cd src/main
go run mrmaster.go pg-being_ernest.txt pg-grimm.txt
  • Start some workers with a mapreduce plugin (eg wordcounter)
cd src/main
# compile plugin
go build -buildmode=plugin ../mrapps/wc.go
# run worker
go run mrworker.go wc.so

Test
cd src/main
sh test-mr.sh

TODO
This project is not ready for distributed envronment. All the mappers produce the intermediate files in the same directory which reducers can access.
In distributed environment, the reducers will either have to connect to mapper via RPC to access the files or the mappers will have to store files in
a distributed file system (eg HDFS)
About
Implementation of MapReduce in Go
Resources
Readme
Activity
Stars
8 stars
Watchers
1 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();
GitHub - vvksh/mapreduce: Implementation of MapReduce in Go · GitHub
Skip to content
BranchesTags
Latest commit
History
19 Commits
Folders and files
NameName
Last commit message
Last commit date
lab_instruction
lab_instruction
src
src
README.md
README.md
Repository files navigation
Mapreduce
This repo contains mapreduce implementation in Go.
This is first lab in MIT's course on distributed systems 6.824. Please see the lab instruction here. Note that I didn't submit the lab, since I wasn't enrolled.
Overview
We have a master and multiple workers.
  1. User starts a master by specifying number of reducers (nR) and a list of files to run mapReduce on. User also starts multiple workers with custom mapper/reducer funtions
  2. Master stores the list of files the users provided and other state in a master Data structure.
  3. When workers are started, they ping master for task every 5 sec (its synchronous, so worker finishes the task at hand before pinging the master again )
  4. During map phase:

    • master gives worker one file to run map
    • worker runs mapper on each file. for each map task (i-th file), the worker produces bunch of key, values. Each key,value is stored as a json entry in a file called mr-{{i}}-{{r}}.txt where i is the file index and r = hash(key) mod nR. So each map tasks produces at most rn files. So overall after all map tasks are done there will be numFiles * nR files.
  5. During Reduce phase:

    • there will be nR reduce tasks. For each r-th task, master gives worker a reduce task with ID = r and a list of all files which have key, values such that r = hash(key) mod nR namely the files
mr-*-{{r}}.txt.
    • the output of each reduce task will be in a file name mr-out-r
Handling worker crash
The master will start a goroutine to monitor state of tasks. It checks if any tasks are not marked DONE within 10 secs. If so, they are marked FAILED and eligible for reassignment.
One of the map reduce apps is a crash.go which crashes or runs slow with certain probabilities. The test file src/main/test-mr.sh produces scenario to run the crash scenario.
How to run
  • Start master with a list of files as argument
cd src/main
go run mrmaster.go pg-being_ernest.txt pg-grimm.txt
  • Start some workers with a mapreduce plugin (eg wordcounter)
cd src/main
# compile plugin
go build -buildmode=plugin ../mrapps/wc.go
# run worker
go run mrworker.go wc.so

Test
cd src/main
sh test-mr.sh

TODO
This project is not ready for distributed envronment. All the mappers produce the intermediate files in the same directory which reducers can access.
In distributed environment, the reducers will either have to connect to mapper via RPC to access the files or the mappers will have to store files in
a distributed file system (eg HDFS)
About
Implementation of MapReduce in Go
Resources
Readme
Activity
Stars
8 stars
Watchers
1 watching
Forks
3 forks
Report repository
Releases
Packages
Used by
Contributors
Languages