simplify dist by merging sql-asm-memory-growth.js into sql-asm.js

[kaizhu256]

Co-authored-by: Yan Li yanli0303@outlook.com
Co-authored-by: kai zhu kaizhu256@gmail.com

this implements suggestion #239 (comment)

removed or edited all references to memory-growth using command git grep -i 'memory.*growth'

[lovasoa]

What is the performance impact ?

[kaizhu256]

good question. the following benchmark indicates asm2 is 1-6% slower running queries on a 60mb database:

// asm2 = LEGACY_VM_SUPPORT + ALLOW_MEMORY_GROWTH
// used build-artifacts from https://github.com/sql-js/sql.js/actions/runs/320372611 for asm2 binaries
// used build-artifacts from https://github.com/sql-js/sql.js/actions/runs/316181408 for asm, wasm binaries
loading 60mb database in chrome
sampled 10 times
asm2: 20.6 +/- 2.2 ms ( 5% slower than asm)
asm : 19.7 +/- 1.9 ms
wasm: 15.4 +/- 0.3 ms (22% faster than asm)
copy all tables in 60mb database in chrome
sampled 5 times
asm2: 15798 +/- 47 ms ( 1% slower than asm)
asm : 15610 +/- 152 ms
wasm: 3384 +/- 47 ms (360% faster than asm)
fts-query with joins in chrome
sampled 5 times
asm2: 40464 +/- 115 ms ( 6% slower than asm)
asm : 38171 +/- 220 ms
wasm: 13942 +/- 216 ms (170% faster than asm)

you can emulate above result in following steps:

1. download 60mb enron-email database from https://github.com/kaizhu256/sqlite-enron

2. load database into modified sql.js demos below:

   • https://kaizhu256.github.io/sql.js/examples/GUI.asm2/
   • https://kaizhu256.github.io/sql.js/examples/GUI.asm/
   • https://kaizhu256.github.io/sql.js/examples/GUI.wasm/

3. run following sql-queries used in benchmarks:

copy all tables in 60mb database

-- create database-schemaCREATETABLE `employeelist2` (
`eid`integerNOT NULLPRIMARY KEY AUTOINCREMENT
, `firstName`varchar(31) NOT NULL DEFAULT ''
, `lastName`varchar(31) NOT NULL DEFAULT ''
, `Email_id`varchar(31) NOT NULL DEFAULT ''
, `Email2`varchar(31) DEFAULT NULL
, `Email3`varchar(31) DEFAULT NULL
, `EMail4`varchar(31) DEFAULT NULL
, `folder`varchar(31) NOT NULL DEFAULT ''
, `status`varchar(50) DEFAULT NULL
, UNIQUE (`Email_id`)
);
CREATETABLE `message2` (
`mid`integerNOT NULL DEFAULT '0'
, `sender`varchar(127) NOT NULL DEFAULT ''
, `date` datetime DEFAULT NULL
, `message_id`varchar(127) DEFAULT NULL-- , `subject` text-- , `body` text
, `folder`varchar(127) NOT NULL DEFAULT ''
, PRIMARY KEY (`mid`)
);
CREATETABLE `recipientinfo2` (
`rid`integerNOT NULL DEFAULT '0'
, `mid`integerNOT NULL DEFAULT '0'
, `rtype`text DEFAULT NULL
, `rvalue`varchar(127) DEFAULT NULL
, `dater` datetime DEFAULT NULL
, PRIMARY KEY (`rid`)
, FOREIGN KEY (mid) REFERENCES message2(mid)
);
CREATETABLE `referenceinfo2` (
`rfid`integerNOT NULL DEFAULT '0'
, `mid`integerNOT NULL DEFAULT '0'
, `reference`text
, PRIMARY KEY (`rfid`)
, FOREIGN KEY (mid) REFERENCES message2(mid)
);
CREATEINDEX "idx_message_sender2" ON"message2" (`sender`);
CREATE VIRTUAL TABLE messages_ft2 USING fts4(subject, body);
-- insert tables-- employeelist2-- message2-- recipientinfo2-- referenceinfo2-- message_ftSELECT'employeelist', COUNT(*) FROM employeelist;
SELECT'message', COUNT(*) FROM message;
SELECT'recipientinfo', COUNT(*) FROM recipientinfo;
SELECT'referenceinfo', COUNT(*) FROM referenceinfo;
SELECT'messages_ft', COUNT(*) FROM messages_ft;
INSERT INTO message2 SELECT*FROM
(
SELECTmessage.mid,
message.sender,
message.date,
message.message_id,
-- message.subject,-- message.body,message.folderFROM
(
SELECT DISTINCTmessage.midFROM message
INNER JOIN recipientinfo ONrecipientinfo.mid=message.midINNER JOIN referenceinfo ONreferenceinfo.mid=message.mid
) AS tmp1
INNER JOIN message ONmessage.mid=tmp1.midORDER BYtmp1.midLIMIT10000
) AS tmp1;
INSERT INTO employeelist2 SELECT*FROM employeelist;
INSERT INTO recipientinfo2 SELECTtmp1.rid,
tmp1.mid,
tmp1.rtype,
tmp1.rvalue,
tmp1.daterFROM message2
LEFT JOIN recipientinfo AS tmp1 ONtmp1.mid=message2.midORDER BYtmp1.rid;
INSERT INTO referenceinfo2 SELECTtmp1.rfid,
tmp1.mid,
tmp1.referenceFROM message2
LEFT JOIN referenceinfo AS tmp1 ONtmp1.mid=message2.midORDER BYtmp1.rfid;
INSERT INTO messages_ft2 (
rowid, subject, body
)
SELECTtmp1.rowid, tmp1.subject, tmp1.bodyFROM message2
INNER JOIN messages_ft AS tmp1 ONtmp1.rowid=message2.midORDER BYtmp1.rowid;
VACUUM;
SELECT'employeelist2', COUNT(*) FROM employeelist2;
SELECT'message2', COUNT(*) FROM message2;
SELECT'recipientinfo2', COUNT(*) FROM recipientinfo2;
SELECT'referenceinfo2', COUNT(*) FROM referenceinfo2;
SELECT'messages_ft2', COUNT(*) FROM messages_ft2;
--*/

fts-query with joins

SELECT*FROM messages_ft2
INNER JOIN recipientinfo2 ONrecipientinfo2.mid=messages_ft2.rowidINNER JOIN referenceinfo2 ONreferenceinfo2.mid=messages_ft2.rowidWHERE body MATCH 'energy NEAR oil';

[kaizhu256]

also the difference in size is < 1%:

 asm asm2
sql-asm-debug.js 10,865,349 10,896,696 bytes
worker.sql-asm-debug.js 10,867,832 10,899,179 bytes
sql-asm.js 2,383,575 2,389,992 bytes
worker.sql-asm.js 2,386,058 2,392,475 bytes

[lovasoa]

If we want to be able to merge this without breaking backwards compatibility, we can first add -s LEGACY_VM_SUPPORT=1 and -s ALLOW_MEMORY_GROWTH=1 into the asmjs version, and add a deprecation warning to the memory growth version.

On the next major version bump, we can delete the memory growth version.

[kaizhu256]

k. would updated patch with following be ok?

[kaizhu256]

latest-patch removes redundant npm-tests on sql-asm-memory-growth.js, since its now essentially a carbon-copy of sql-asm.js (plus a deprecation-warning header).

this simplifies sunsetting sql-asm-memory-growth.js in some future-release by simply deleting the following lines in /Makefile:

--- a/Makefile+++ b/Makefile@@ -101,9 +101,6 @@ dist/sql-wasm.js: $(BITCODE_FILES) $(OUTPUT_WRAPPER_FILES) $(SOURCE_API_FILES) $
mv $@ out/tmp-raw.js
cat src/shell-pre.js out/tmp-raw.js src/shell-post.js > $@
rm out/tmp-raw.js
- # TODO - remove code below in future releases to sunset sql-asm-memory-growth.js- printf 'console.error(\n "\\n\\nDEPRECATION WARNING.\\n"\n + "sql-asm-memory-growth.js will be removed in future releases.\\n"\n + "Use sql-asm.js instead, which now includes memory-growth support.\\n\\n"\n);\n' > dist/sql-asm-memory-growth.js- cat $@ >> dist/sql-asm-memory-growth.js
# Web worker API
.PHONY: worker

if you still want npm-tests on sql-asm-memory-growth.js i can revert this patch