Restrict upload for dangerous file types

[harminius]

Resolveshttps://github.com/MerginMaps/server-private/issues/2732

Problem:
We allow users to upload any file, even the (potentially) malicious ones 🏴‍☠️

See tickets for more details and the Pentest report

Solution:
We are more confident with using blacklists. 🛡️

Validate extension during project push - aka push_start. Abort the upload if it includes files with blacklisted extensions.
Do not rely solely on the extension check. Determine the file type from its header and block blacklisted types - once we have the file on our filesystem (in push_finish).

Zipping the unsupported makes it possible to upload it.

E.g. block .exe

Extension renaming of the unsupported file doesn't help ⛔

Potential whitelists are in the comment

Do not allow to sync files with extensions unless whitelisted

Whitelist approach is suggested in this PR as it is considered to be safer. However, using blacklists is also a way, see this comment for suggested blacklists.

[coveralls]

Pull Request Test Coverage Report for Build 13318729670

Details

• 50 of 50(100.0%) changed or added relevant lines in 3 files are covered.
• 49 unchanged lines in 3 files lost coverage.
• Overall coverage increased (+0.9%) to 93.275%

---

Files with Coverage Reduction	New Missed Lines	%
server/mergin/sync/utils.py	1	84.13%
server/mergin/commands.py	12	31.71%
server/mergin/app.py	36	74.52%

Totals
Change from base Build 13240347753:	0.9%
Covered Lines:	6755
Relevant Lines:	7242

---

💛 - Coveralls

[harminius]

Current approach is (in web security recommended) whitelist.
If we are afraid to be too restrictive we could use blackilists.
Blocking logic would stay the same.

Here's the suggestion:

FORBIDDEN_EXTENSIONS = {
".ade", ".adp", ".app", ".appcontent-ms", ".application", ".appref-ms", ".asp", ".aspx", ".asx", ".bas", ".bat", ".bgi", ".cab", ".cdxml", ".cer", ".chm", ".cmd", ".cnt", ".com", ".cpl", ".crt", ".csh", ".der", ".diagcab", ".dll", ".drv", ".exe", ".fxp", ".gadget", ".grp", ".hlp", ".hpj", ".hta", ".htc", ".htaccess", ".htpasswd", ".inf", ".ins", ".iso", ".isp", ".its", ".jar", ".jnlp", ".js", ".jse", ".jsp", ".ksh", ".lnk", ".mad", ".maf", ".mag", ".mam", ".maq", ".mar", ".mas", ".mat", ".mau", ".mav", ".maw", ".mcf", ".mda", ".mdb", ".mde", ".mdt", ".mdw", ".mdz", ".msc", ".mht", ".mhtml", ".msh", ".msh1", ".msh2", ".mshxml", ".msh1xml", ".msh2xml", ".msi", ".msp", ".mst", ".msu", ".ops", ".osd", ".pcd", ".pif", ".pl", ".plg", ".prf", ".prg", ".printerexport", ".ps1", ".ps1xml", ".ps2", ".ps2xml", ".psc1", ".psc2", ".psd1", ".psdm1", ".pssc", ".pst", ".py", ".pyc", ".pyo", ".pyw", ".pyz", ".pyzw", ".reg", ".scf", ".scr", ".sct", ".settingcontent-ms", ".sh", ".shb", ".shs", ".sys", ".theme", ".tmp", ".torrent", ".url", ".vb", ".vbe", ".vbp", ".vbs", ".vhd", ".vhdx", ".vsmacros", ".vsw", ".webpnp", ".website", ".ws", ".wsb", ".wsc", ".wsf", ".wsh", ".xbap", ".xll", ".xnk"
}

FORBIDDEN_MIME_TYPES = {
"application/x-msdownload",
"application/x-sh",
"application/x-bat",
"application/x-msdos-program",
"application/x-dosexec",
"application/x-csh",
"application/x-perl",
"application/javascript",
"application/x-python-code",
"application/x-ruby",
"application/java-archive",
"application/vnd.ms-cab-compressed",
"application/x-ms-shortcut",
"application/vnd.microsoft.portable-executable",
"application/x-ms-installer",
"application/x-ms-application",
"application/x-ms-wim",
"text/x-shellscript",
}

[MarcelGeo]

Current approach is (in web security recommended) whitelist. If we are afraid to be too restrictive we could use blackilists. Blocking logic would stay the same.

Here's the suggestion:

FORBIDDEN_EXTENSIONS = {
".ade", ".adp", ".app", ".appcontent-ms", ".application", ".appref-ms", ".asp", ".aspx", ".asx", ".bas", ".bat", ".bgi", ".cab", ".cdxml", ".cer", ".chm", ".cmd", ".cnt", ".com", ".cpl", ".crt", ".csh", ".der", ".diagcab", ".dll", ".drv", ".exe", ".fxp", ".gadget", ".grp", ".hlp", ".hpj", ".hta", ".htc", ".htaccess", ".htpasswd", ".img", ".inf", ".ins", ".iso", ".isp", ".its", ".jar", ".jnlp", ".js", ".jse", ".jsp", ".ksh", ".lnk", ".mad", ".maf", ".mag", ".mam", ".maq", ".mar", ".mas", ".mat", ".mau", ".mav", ".maw", ".mcf", ".mda", ".mdb", ".mde", ".mdt", ".mdw", ".mdz", ".msc", ".mht", ".mhtml", ".msh", ".msh1", ".msh2", ".mshxml", ".msh1xml", ".msh2xml", ".msi", ".msp", ".mst", ".msu", ".ops", ".osd", ".pcd", ".pif", ".pl", ".plg", ".prf", ".prg", ".printerexport", ".ps1", ".ps1xml", ".ps2", ".ps2xml", ".psc1", ".psc2", ".psd1", ".psdm1", ".pssc", ".pst", ".py", ".pyc", ".pyo", ".pyw", ".pyz", ".pyzw", ".reg", ".scf", ".scr", ".sct", ".settingcontent-ms", ".sh", ".shb", ".shs", ".sys", ".theme", ".tmp", ".torrent", ".url", ".vb", ".vbe", ".vbp", ".vbs", ".vhd", ".vhdx", ".vsmacros", ".vsw", ".webpnp", ".website", ".ws", ".wsb", ".wsc", ".wsf", ".wsh", ".xbap", ".xll", ".xnk"
}

FORBIDDEN_MIME_TYPES = {
"application/x-msdownload",
"application/x-sh",
"application/x-bat",
"application/x-msdos-program",
"application/x-dosexec",
"application/x-csh",
"application/x-perl",
"application/javascript",
"application/x-python-code",
"application/x-ruby",
"application/java-archive",
"application/vnd.ms-cab-compressed",
"application/octet-stream",
"application/x-ms-shortcut",
"application/vnd.microsoft.portable-executable",
"application/x-ms-installer",
"application/x-ms-application",
"application/x-ms-wim",
"text/x-shellscript",
}

This looks more sensible for me now.

[harminius]

whitelist:
ALLOWED_EXTENSIONS = {
# Geospatial
# Shapefile components
".shp",
".shx",
".dbf",
".prj",
".cpg",
".qix",
".sbn",
".sbx",
# Vector data formats
".geojson",
".kml",
".kmz",
".gpx",
".dxf",
".svg",
".gpkg",
".json",
# Raster data formats
".tif",
".tiff",
".geotiff",
".asc",
".vrt",
".grd",
".img",
".adf",
# Point cloud data formats
".las",
".laz",
".ply",
".xyz",
".e57",
".pcd",
# Database and container formats
".mbtiles",
".sqlite",
".gpkg",
# Geospatial metadata and styles
".sld",
".qml",
".lyr",
".qgz",
".qgs",
# Other specialized formats
".hdf",
".hdf5",
".netcdf",
".nc",
".dem",
".dt2",
".dt0",
".map",
".tab",
".mif",
".mid",
# Images
".jpg",
".jpeg",
".png",
".bmp",
".gif",
".heic",
".webp",
".tif",
".tiff",
# Text documents
".pdf",
".doc",
".docx",
".odt",
".rtf",
".txt",
# Others
".zip",
}

ALLOWED_MIME_TYPES = {
"application/x-shapefile",
"application/x-dbf",
"text/plain",
"application/octet-stream",
"application/geo+json",
"application/vnd.google-earth.kml+xml",
"application/vnd.google-earth.kmz",
"application/gpx+xml",
"application/geopackage+sqlite3",
"application/vnd.sqlite3",
"application/json",
"text/xml",
"text/html",
"application/vnd.mapbox-vector-tile",
"application/x-sqlite3",
"application/vnd.ogc.sld+xml",
"application/xml",
"application/x-qgis",
"application/x-hdf",
"application/x-hdf5",
"application/x-netcdf",
"application/pdf",
"application/msword",
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
"application/vnd.oasis.opendocument.text",
"application/rtf",
"text/plain",
"application/zip",
}

[tomasMizera]

The platform is getting more and more stable 🚀

Let's make sure we are ready to quickly adjust the list in case we accidentally block something we should not have blocked.

[tomasMizera]

How about

Suggested change

	abort(400, f"File '{item.path}' contains invalid characters.")
	abort(400, f"Unsupported file name detected: {item.path}")

to make it clear it relates to the file name(path), not the content of the file

[tomasMizera]

I'd suggest

Suggested change

	abort(400, f"Unsupported extension of '{item.path}' file.")
	abort(400, f"Unsupported file type detected: {item.path}")

File type is easier to understand than extension. While it is obvious to us, end users might not know what that refers to :)

[harminius]

Shall we add also a hint what to do to unblock upload? Please remove the file from the upload folder or try zipping the file before uploading.
""

[tomasMizera]

Yeah, good idea! 👍🏻 How about Please remove the file or try compressing it into a ZIP file before uploading.?

[tomasMizera]

Similarly here:

Suggested change

	abort(400, f"Unsupported file type of '{f.path}' file.")
	abort(400, f"Unsupported file type detected: {f.path}")