Latest commit

History

21 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

Discogs-dump-parser

A lightweight parser of Discogs data dumps

This software is released under a Creative Commons "CC0" Public Domain Dedication; see https://creativecommons.org/publicdomain/zero/1.0/ for more info.

Discogs artist, label, release, and master release data is publicly available in huge XML files produced monthly and made available at https://data.discogs.com/ for anyone to download.

This Python script parses the Discogs release data using the ultrafast cElementTree API which comes with Python 2.5 and up. The script automatically handles compressed and uncompressed data, and the old style of dump which had no root element.

Demo

To try it out, get one of the release data dump files and just run the script, passing the dump file path as the only argument. For example, if the script and the dump file named discogs_20191101_releases.xml.gz are in the same directory:

python parse_discogs_dump.py discogs_20191101_releases.xml.gz

By default, a dot will be printed to the screen for every 1000 'release' elements read. At the end it tells you how much time it took. If you interrupt it, it tells you the last release ID it saw. The parsed data is mostly ignored; the idea is just to successfully read the XML, building a temporary tree for each 'release' element.

As of 2018, on my 3.1 GHz Intel Core i5-2400 system (using only 1 of 4 cores), it takes 61 minutes to plow through the 6.0 GB gzipped release data XML and print the dots, yet it only ever uses about 17 MB of memory. It could run faster if a temporary tree was not built and discarded for each 'release', but I feel it is a better benchmark this way.

If you uncomment one line of code near the end, then instead of a dot, you can get a complete XML fragment for every 1000th release element read.

Customization

There is no need to modify parse_discogs_dump.py directly. You can write your own code to handle each 'release' element, which will be an instance of ElementTree.Element. Your code needs to just do the following:

  1. Import ElementProcessor and process_dump_file() from parse_discogs_dump.py.
  2. Define a subclass of ElementProcessor with, at a minimum, a process() method to handle each 'release' element (which will be an instance of ElementTree.Element) in whatever way you want.
  3. Pass the dump file path and an instance of your subclass to process_dump_file().

For example: find_invalid_release_dates.py is a script which does exactly those things. It can be run like this:

python find_invalid_release_dates.py discogs_20191101_releases.xml.gz > report.txt

Every time it finds a non-empty release date which does not match the patterns #### or ####-##-## with a non-zero month value, it will print a dot to the screen, and the output file report.txt will get a line like this:

https://www.discogs.com/release/41748 - release date is "?"

Error handling

Some Discogs dump files contain errors in the XML. If you get an error message about the XML not being well-formed, you will have to fix the dump file. For example, you might need to remove the control characters which are forbidden in XML:

gzcat discogs_20080309_releases.xml.gz | tr -d '\1\2\3\4\5\6\7\10\13\14\16\17\20\21\22\23\24\25\26\27\30\31\32\33\34\35\36\37\177\200\201\202\203\204\206\207\210\211\212\213\214\215\216\217\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237' | gzip -9 > discogs_20080309_releases.fixed.xml.gz

Contact

I am user mjb on Discogs. Feel free to contact me there via private message, or in the API forum.

About

A lightweight parser of Discogs data dumps

Resources

Stars

25 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Latest commit

History

21 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

Discogs-dump-parser

A lightweight parser of Discogs data dumps

This software is released under a Creative Commons "CC0" Public Domain Dedication; see https://creativecommons.org/publicdomain/zero/1.0/ for more info.

Discogs artist, label, release, and master release data is publicly available in huge XML files produced monthly and made available at https://data.discogs.com/ for anyone to download.

This Python script parses the Discogs release data using the ultrafast cElementTree API which comes with Python 2.5 and up. The script automatically handles compressed and uncompressed data, and the old style of dump which had no root element.

Demo

To try it out, get one of the release data dump files and just run the script, passing the dump file path as the only argument. For example, if the script and the dump file named discogs_20191101_releases.xml.gz are in the same directory:

python parse_discogs_dump.py discogs_20191101_releases.xml.gz

By default, a dot will be printed to the screen for every 1000 'release' elements read. At the end it tells you how much time it took. If you interrupt it, it tells you the last release ID it saw. The parsed data is mostly ignored; the idea is just to successfully read the XML, building a temporary tree for each 'release' element.

As of 2018, on my 3.1 GHz Intel Core i5-2400 system (using only 1 of 4 cores), it takes 61 minutes to plow through the 6.0 GB gzipped release data XML and print the dots, yet it only ever uses about 17 MB of memory. It could run faster if a temporary tree was not built and discarded for each 'release', but I feel it is a better benchmark this way.

If you uncomment one line of code near the end, then instead of a dot, you can get a complete XML fragment for every 1000th release element read.

Customization

There is no need to modify parse_discogs_dump.py directly. You can write your own code to handle each 'release' element, which will be an instance of ElementTree.Element. Your code needs to just do the following:

  1. Import ElementProcessor and process_dump_file() from parse_discogs_dump.py.
  2. Define a subclass of ElementProcessor with, at a minimum, a process() method to handle each 'release' element (which will be an instance of ElementTree.Element) in whatever way you want.
  3. Pass the dump file path and an instance of your subclass to process_dump_file().

For example: find_invalid_release_dates.py is a script which does exactly those things. It can be run like this:

python find_invalid_release_dates.py discogs_20191101_releases.xml.gz > report.txt

Every time it finds a non-empty release date which does not match the patterns #### or ####-##-## with a non-zero month value, it will print a dot to the screen, and the output file report.txt will get a line like this:

https://www.discogs.com/release/41748 - release date is "?"

Error handling

Some Discogs dump files contain errors in the XML. If you get an error message about the XML not being well-formed, you will have to fix the dump file. For example, you might need to remove the control characters which are forbidden in XML:

gzcat discogs_20080309_releases.xml.gz | tr -d '\1\2\3\4\5\6\7\10\13\14\16\17\20\21\22\23\24\25\26\27\30\31\32\33\34\35\36\37\177\200\201\202\203\204\206\207\210\211\212\213\214\215\216\217\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237' | gzip -9 > discogs_20080309_releases.fixed.xml.gz

Contact

I am user mjb on Discogs. Feel free to contact me there via private message, or in the API forum.

About

A lightweight parser of Discogs data dumps

Resources

Stars

25 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Latest commit

History

21 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

Discogs-dump-parser

A lightweight parser of Discogs data dumps

This software is released under a Creative Commons "CC0" Public Domain Dedication; see https://creativecommons.org/publicdomain/zero/1.0/ for more info.

Discogs artist, label, release, and master release data is publicly available in huge XML files produced monthly and made available at https://data.discogs.com/ for anyone to download.

This Python script parses the Discogs release data using the ultrafast cElementTree API which comes with Python 2.5 and up. The script automatically handles compressed and uncompressed data, and the old style of dump which had no root element.

Demo

To try it out, get one of the release data dump files and just run the script, passing the dump file path as the only argument. For example, if the script and the dump file named discogs_20191101_releases.xml.gz are in the same directory:

python parse_discogs_dump.py discogs_20191101_releases.xml.gz

By default, a dot will be printed to the screen for every 1000 'release' elements read. At the end it tells you how much time it took. If you interrupt it, it tells you the last release ID it saw. The parsed data is mostly ignored; the idea is just to successfully read the XML, building a temporary tree for each 'release' element.

As of 2018, on my 3.1 GHz Intel Core i5-2400 system (using only 1 of 4 cores), it takes 61 minutes to plow through the 6.0 GB gzipped release data XML and print the dots, yet it only ever uses about 17 MB of memory. It could run faster if a temporary tree was not built and discarded for each 'release', but I feel it is a better benchmark this way.

If you uncomment one line of code near the end, then instead of a dot, you can get a complete XML fragment for every 1000th release element read.

Customization

There is no need to modify parse_discogs_dump.py directly. You can write your own code to handle each 'release' element, which will be an instance of ElementTree.Element. Your code needs to just do the following:

  1. Import ElementProcessor and process_dump_file() from parse_discogs_dump.py.
  2. Define a subclass of ElementProcessor with, at a minimum, a process() method to handle each 'release' element (which will be an instance of ElementTree.Element) in whatever way you want.
  3. Pass the dump file path and an instance of your subclass to process_dump_file().

For example: find_invalid_release_dates.py is a script which does exactly those things. It can be run like this:

python find_invalid_release_dates.py discogs_20191101_releases.xml.gz > report.txt

Every time it finds a non-empty release date which does not match the patterns #### or ####-##-## with a non-zero month value, it will print a dot to the screen, and the output file report.txt will get a line like this:

https://www.discogs.com/release/41748 - release date is "?"

Error handling

Some Discogs dump files contain errors in the XML. If you get an error message about the XML not being well-formed, you will have to fix the dump file. For example, you might need to remove the control characters which are forbidden in XML:

gzcat discogs_20080309_releases.xml.gz | tr -d '\1\2\3\4\5\6\7\10\13\14\16\17\20\21\22\23\24\25\26\27\30\31\32\33\34\35\36\37\177\200\201\202\203\204\206\207\210\211\212\213\214\215\216\217\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237' | gzip -9 > discogs_20080309_releases.fixed.xml.gz

Contact

I am user mjb on Discogs. Feel free to contact me there via private message, or in the API forum.

About

A lightweight parser of Discogs data dumps

Resources

Stars

25 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Latest commit

History

21 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

Discogs-dump-parser

A lightweight parser of Discogs data dumps

This software is released under a Creative Commons "CC0" Public Domain Dedication; see https://creativecommons.org/publicdomain/zero/1.0/ for more info.

Discogs artist, label, release, and master release data is publicly available in huge XML files produced monthly and made available at https://data.discogs.com/ for anyone to download.

This Python script parses the Discogs release data using the ultrafast cElementTree API which comes with Python 2.5 and up. The script automatically handles compressed and uncompressed data, and the old style of dump which had no root element.

Demo

To try it out, get one of the release data dump files and just run the script, passing the dump file path as the only argument. For example, if the script and the dump file named discogs_20191101_releases.xml.gz are in the same directory:

python parse_discogs_dump.py discogs_20191101_releases.xml.gz

By default, a dot will be printed to the screen for every 1000 'release' elements read. At the end it tells you how much time it took. If you interrupt it, it tells you the last release ID it saw. The parsed data is mostly ignored; the idea is just to successfully read the XML, building a temporary tree for each 'release' element.

As of 2018, on my 3.1 GHz Intel Core i5-2400 system (using only 1 of 4 cores), it takes 61 minutes to plow through the 6.0 GB gzipped release data XML and print the dots, yet it only ever uses about 17 MB of memory. It could run faster if a temporary tree was not built and discarded for each 'release', but I feel it is a better benchmark this way.

If you uncomment one line of code near the end, then instead of a dot, you can get a complete XML fragment for every 1000th release element read.

Customization

There is no need to modify parse_discogs_dump.py directly. You can write your own code to handle each 'release' element, which will be an instance of ElementTree.Element. Your code needs to just do the following:

  1. Import ElementProcessor and process_dump_file() from parse_discogs_dump.py.
  2. Define a subclass of ElementProcessor with, at a minimum, a process() method to handle each 'release' element (which will be an instance of ElementTree.Element) in whatever way you want.
  3. Pass the dump file path and an instance of your subclass to process_dump_file().

For example: find_invalid_release_dates.py is a script which does exactly those things. It can be run like this:

python find_invalid_release_dates.py discogs_20191101_releases.xml.gz > report.txt

Every time it finds a non-empty release date which does not match the patterns #### or ####-##-## with a non-zero month value, it will print a dot to the screen, and the output file report.txt will get a line like this:

https://www.discogs.com/release/41748 - release date is "?"

Error handling

Some Discogs dump files contain errors in the XML. If you get an error message about the XML not being well-formed, you will have to fix the dump file. For example, you might need to remove the control characters which are forbidden in XML:

gzcat discogs_20080309_releases.xml.gz | tr -d '\1\2\3\4\5\6\7\10\13\14\16\17\20\21\22\23\24\25\26\27\30\31\32\33\34\35\36\37\177\200\201\202\203\204\206\207\210\211\212\213\214\215\216\217\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237' | gzip -9 > discogs_20080309_releases.fixed.xml.gz

Contact

I am user mjb on Discogs. Feel free to contact me there via private message, or in the API forum.

About

A lightweight parser of Discogs data dumps

Resources

Stars

25 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Latest commit

History

21 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

Discogs-dump-parser

A lightweight parser of Discogs data dumps

This software is released under a Creative Commons "CC0" Public Domain Dedication; see https://creativecommons.org/publicdomain/zero/1.0/ for more info.

Discogs artist, label, release, and master release data is publicly available in huge XML files produced monthly and made available at https://data.discogs.com/ for anyone to download.

This Python script parses the Discogs release data using the ultrafast cElementTree API which comes with Python 2.5 and up. The script automatically handles compressed and uncompressed data, and the old style of dump which had no root element.

Demo

To try it out, get one of the release data dump files and just run the script, passing the dump file path as the only argument. For example, if the script and the dump file named discogs_20191101_releases.xml.gz are in the same directory:

python parse_discogs_dump.py discogs_20191101_releases.xml.gz

By default, a dot will be printed to the screen for every 1000 'release' elements read. At the end it tells you how much time it took. If you interrupt it, it tells you the last release ID it saw. The parsed data is mostly ignored; the idea is just to successfully read the XML, building a temporary tree for each 'release' element.

As of 2018, on my 3.1 GHz Intel Core i5-2400 system (using only 1 of 4 cores), it takes 61 minutes to plow through the 6.0 GB gzipped release data XML and print the dots, yet it only ever uses about 17 MB of memory. It could run faster if a temporary tree was not built and discarded for each 'release', but I feel it is a better benchmark this way.

If you uncomment one line of code near the end, then instead of a dot, you can get a complete XML fragment for every 1000th release element read.

Customization

There is no need to modify parse_discogs_dump.py directly. You can write your own code to handle each 'release' element, which will be an instance of ElementTree.Element. Your code needs to just do the following:

  1. Import ElementProcessor and process_dump_file() from parse_discogs_dump.py.
  2. Define a subclass of ElementProcessor with, at a minimum, a process() method to handle each 'release' element (which will be an instance of ElementTree.Element) in whatever way you want.
  3. Pass the dump file path and an instance of your subclass to process_dump_file().

For example: find_invalid_release_dates.py is a script which does exactly those things. It can be run like this:

python find_invalid_release_dates.py discogs_20191101_releases.xml.gz > report.txt

Every time it finds a non-empty release date which does not match the patterns #### or ####-##-## with a non-zero month value, it will print a dot to the screen, and the output file report.txt will get a line like this:

https://www.discogs.com/release/41748 - release date is "?"

Error handling

Some Discogs dump files contain errors in the XML. If you get an error message about the XML not being well-formed, you will have to fix the dump file. For example, you might need to remove the control characters which are forbidden in XML:

gzcat discogs_20080309_releases.xml.gz | tr -d '\1\2\3\4\5\6\7\10\13\14\16\17\20\21\22\23\24\25\26\27\30\31\32\33\34\35\36\37\177\200\201\202\203\204\206\207\210\211\212\213\214\215\216\217\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237' | gzip -9 > discogs_20080309_releases.fixed.xml.gz

Contact

I am user mjb on Discogs. Feel free to contact me there via private message, or in the API forum.

About

A lightweight parser of Discogs data dumps

Resources

Stars

25 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Latest commit

History

21 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

Discogs-dump-parser

A lightweight parser of Discogs data dumps

This software is released under a Creative Commons "CC0" Public Domain Dedication; see https://creativecommons.org/publicdomain/zero/1.0/ for more info.

Discogs artist, label, release, and master release data is publicly available in huge XML files produced monthly and made available at https://data.discogs.com/ for anyone to download.

This Python script parses the Discogs release data using the ultrafast cElementTree API which comes with Python 2.5 and up. The script automatically handles compressed and uncompressed data, and the old style of dump which had no root element.

Demo

To try it out, get one of the release data dump files and just run the script, passing the dump file path as the only argument. For example, if the script and the dump file named discogs_20191101_releases.xml.gz are in the same directory:

python parse_discogs_dump.py discogs_20191101_releases.xml.gz

By default, a dot will be printed to the screen for every 1000 'release' elements read. At the end it tells you how much time it took. If you interrupt it, it tells you the last release ID it saw. The parsed data is mostly ignored; the idea is just to successfully read the XML, building a temporary tree for each 'release' element.

As of 2018, on my 3.1 GHz Intel Core i5-2400 system (using only 1 of 4 cores), it takes 61 minutes to plow through the 6.0 GB gzipped release data XML and print the dots, yet it only ever uses about 17 MB of memory. It could run faster if a temporary tree was not built and discarded for each 'release', but I feel it is a better benchmark this way.

If you uncomment one line of code near the end, then instead of a dot, you can get a complete XML fragment for every 1000th release element read.

Customization

There is no need to modify parse_discogs_dump.py directly. You can write your own code to handle each 'release' element, which will be an instance of ElementTree.Element. Your code needs to just do the following:

  1. Import ElementProcessor and process_dump_file() from parse_discogs_dump.py.
  2. Define a subclass of ElementProcessor with, at a minimum, a process() method to handle each 'release' element (which will be an instance of ElementTree.Element) in whatever way you want.
  3. Pass the dump file path and an instance of your subclass to process_dump_file().

For example: find_invalid_release_dates.py is a script which does exactly those things. It can be run like this:

python find_invalid_release_dates.py discogs_20191101_releases.xml.gz > report.txt

Every time it finds a non-empty release date which does not match the patterns #### or ####-##-## with a non-zero month value, it will print a dot to the screen, and the output file report.txt will get a line like this:

https://www.discogs.com/release/41748 - release date is "?"

Error handling

Some Discogs dump files contain errors in the XML. If you get an error message about the XML not being well-formed, you will have to fix the dump file. For example, you might need to remove the control characters which are forbidden in XML:

gzcat discogs_20080309_releases.xml.gz | tr -d '\1\2\3\4\5\6\7\10\13\14\16\17\20\21\22\23\24\25\26\27\30\31\32\33\34\35\36\37\177\200\201\202\203\204\206\207\210\211\212\213\214\215\216\217\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237' | gzip -9 > discogs_20080309_releases.fixed.xml.gz

Contact

I am user mjb on Discogs. Feel free to contact me there via private message, or in the API forum.

About

A lightweight parser of Discogs data dumps

Resources

Stars

25 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Latest commit

History

21 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

Discogs-dump-parser

A lightweight parser of Discogs data dumps

This software is released under a Creative Commons "CC0" Public Domain Dedication; see https://creativecommons.org/publicdomain/zero/1.0/ for more info.

Discogs artist, label, release, and master release data is publicly available in huge XML files produced monthly and made available at https://data.discogs.com/ for anyone to download.

This Python script parses the Discogs release data using the ultrafast cElementTree API which comes with Python 2.5 and up. The script automatically handles compressed and uncompressed data, and the old style of dump which had no root element.

Demo

To try it out, get one of the release data dump files and just run the script, passing the dump file path as the only argument. For example, if the script and the dump file named discogs_20191101_releases.xml.gz are in the same directory:

python parse_discogs_dump.py discogs_20191101_releases.xml.gz

By default, a dot will be printed to the screen for every 1000 'release' elements read. At the end it tells you how much time it took. If you interrupt it, it tells you the last release ID it saw. The parsed data is mostly ignored; the idea is just to successfully read the XML, building a temporary tree for each 'release' element.

As of 2018, on my 3.1 GHz Intel Core i5-2400 system (using only 1 of 4 cores), it takes 61 minutes to plow through the 6.0 GB gzipped release data XML and print the dots, yet it only ever uses about 17 MB of memory. It could run faster if a temporary tree was not built and discarded for each 'release', but I feel it is a better benchmark this way.

If you uncomment one line of code near the end, then instead of a dot, you can get a complete XML fragment for every 1000th release element read.

Customization

There is no need to modify parse_discogs_dump.py directly. You can write your own code to handle each 'release' element, which will be an instance of ElementTree.Element. Your code needs to just do the following:

  1. Import ElementProcessor and process_dump_file() from parse_discogs_dump.py.
  2. Define a subclass of ElementProcessor with, at a minimum, a process() method to handle each 'release' element (which will be an instance of ElementTree.Element) in whatever way you want.
  3. Pass the dump file path and an instance of your subclass to process_dump_file().

For example: find_invalid_release_dates.py is a script which does exactly those things. It can be run like this:

python find_invalid_release_dates.py discogs_20191101_releases.xml.gz > report.txt

Every time it finds a non-empty release date which does not match the patterns #### or ####-##-## with a non-zero month value, it will print a dot to the screen, and the output file report.txt will get a line like this:

https://www.discogs.com/release/41748 - release date is "?"

Error handling

Some Discogs dump files contain errors in the XML. If you get an error message about the XML not being well-formed, you will have to fix the dump file. For example, you might need to remove the control characters which are forbidden in XML:

gzcat discogs_20080309_releases.xml.gz | tr -d '\1\2\3\4\5\6\7\10\13\14\16\17\20\21\22\23\24\25\26\27\30\31\32\33\34\35\36\37\177\200\201\202\203\204\206\207\210\211\212\213\214\215\216\217\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237' | gzip -9 > discogs_20080309_releases.fixed.xml.gz

Contact

I am user mjb on Discogs. Feel free to contact me there via private message, or in the API forum.

About

A lightweight parser of Discogs data dumps

Resources

Stars

25 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages

, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Latest commit

History

21 Commits

Folders and files

NameName
Last commit message
Last commit date

Repository files navigation

Discogs-dump-parser

A lightweight parser of Discogs data dumps

This software is released under a Creative Commons "CC0" Public Domain Dedication; see https://creativecommons.org/publicdomain/zero/1.0/ for more info.

Discogs artist, label, release, and master release data is publicly available in huge XML files produced monthly and made available at https://data.discogs.com/ for anyone to download.

This Python script parses the Discogs release data using the ultrafast cElementTree API which comes with Python 2.5 and up. The script automatically handles compressed and uncompressed data, and the old style of dump which had no root element.

Demo

To try it out, get one of the release data dump files and just run the script, passing the dump file path as the only argument. For example, if the script and the dump file named discogs_20191101_releases.xml.gz are in the same directory:

python parse_discogs_dump.py discogs_20191101_releases.xml.gz

By default, a dot will be printed to the screen for every 1000 'release' elements read. At the end it tells you how much time it took. If you interrupt it, it tells you the last release ID it saw. The parsed data is mostly ignored; the idea is just to successfully read the XML, building a temporary tree for each 'release' element.

As of 2018, on my 3.1 GHz Intel Core i5-2400 system (using only 1 of 4 cores), it takes 61 minutes to plow through the 6.0 GB gzipped release data XML and print the dots, yet it only ever uses about 17 MB of memory. It could run faster if a temporary tree was not built and discarded for each 'release', but I feel it is a better benchmark this way.

If you uncomment one line of code near the end, then instead of a dot, you can get a complete XML fragment for every 1000th release element read.

Customization

There is no need to modify parse_discogs_dump.py directly. You can write your own code to handle each 'release' element, which will be an instance of ElementTree.Element. Your code needs to just do the following:

  1. Import ElementProcessor and process_dump_file() from parse_discogs_dump.py.
  2. Define a subclass of ElementProcessor with, at a minimum, a process() method to handle each 'release' element (which will be an instance of ElementTree.Element) in whatever way you want.
  3. Pass the dump file path and an instance of your subclass to process_dump_file().

For example: find_invalid_release_dates.py is a script which does exactly those things. It can be run like this:

python find_invalid_release_dates.py discogs_20191101_releases.xml.gz > report.txt

Every time it finds a non-empty release date which does not match the patterns #### or ####-##-## with a non-zero month value, it will print a dot to the screen, and the output file report.txt will get a line like this:

https://www.discogs.com/release/41748 - release date is "?"

Error handling

Some Discogs dump files contain errors in the XML. If you get an error message about the XML not being well-formed, you will have to fix the dump file. For example, you might need to remove the control characters which are forbidden in XML:

gzcat discogs_20080309_releases.xml.gz | tr -d '\1\2\3\4\5\6\7\10\13\14\16\17\20\21\22\23\24\25\26\27\30\31\32\33\34\35\36\37\177\200\201\202\203\204\206\207\210\211\212\213\214\215\216\217\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237' | gzip -9 > discogs_20080309_releases.fixed.xml.gz

Contact

I am user mjb on Discogs. Feel free to contact me there via private message, or in the API forum.

About

A lightweight parser of Discogs data dumps

Resources

Stars

25 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages