Skip to content

Shard gl jasmine tests#2933

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
etpinard merged 16 commits into from
Aug 21, 2018
Merged

Shard gl jasmine tests #2933

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
635b643
move @gl in 'describe' to child 'it'
etpinard Aug 20, 2018
53657e7
shard gl jasmine tests
etpinard Aug 20, 2018
02308d0
fixup retry message
etpinard Aug 20, 2018
094e06d
unset flaky in gl2d_click
etpinard Aug 20, 2018
c9a4eb5
try new syntax tests
etpinard Aug 20, 2018
e23ee6e
enforce no @gl + @flaky on same test
etpinard Aug 20, 2018
6d8da96
enforce @flaky tags on 'it' blocks only
etpinard Aug 20, 2018
ba753c4
rm redundant skipped tag
etpinard Aug 21, 2018
8170900
try failOnEmptyTestSuite:false
etpinard Aug 21, 2018
7712507
purge better in-between gl2d tests
etpinard Jul 31, 2018
bdf468e
move flaky tests to jasmine)
etpinard Aug 21, 2018
689671b
add --failFast cli option to karma runner
etpinard Aug 21, 2018
c88bdbb
adapt ci retry karma loop to fail fast on all but last run
etpinard Aug 21, 2018
6c29290
set browserConsoleLog level to 'debug'
etpinard Aug 21, 2018
6daf432
don't bundle lib/index.js before test files
etpinard Aug 21, 2018
006d5d1
fixup - add info about --failFast
etpinard Aug 21, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 21 additions & 8 deletions .circleci/test.sh
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -4,23 +4,31 @@
set +e
set +o pipefail

ROOT=$(dirname $0)/..
EXIT_STATE=0
MAX_AUTO_RETRY=5

log () {
echo -e "\n$1"
}

# inspired by https://unix.stackexchange.com/a/82602
retry () {
local n=0
local n=1

until [ $n -ge $MAX_AUTO_RETRY ]; do
"$@" && break
"$@" --failFast && break
log "run $n of $MAX_AUTO_RETRY failed, trying again ..."
n=$[$n+1]
echo ''
echo run $n of $MAX_AUTO_RETRY failed, trying again ...
echo ''
sleep 15
done

if [ $n -eq $MAX_AUTO_RETRY ]; then
log "one last time, w/o failing fast"
"$@" && n=0

@alexcjohnsonalexcjohnsonAug 21, 2018

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

man, shell scripts are ugly... that said, very nicely done 👍

@etpinardetpinardAug 21, 2018

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was the cleanest solution I could think of 😑

fi

if [ $n -eq $MAX_AUTO_RETRY ]; then
log "all $n runs failed, moving on."
EXIT_STATE=1
fi
}
Expand All@@ -29,13 +37,18 @@ case $1 in

jasmine)
npm run test-jasmine -- --skip-tags=gl,noCI,flaky || EXIT_STATE=$?
retry npm run test-jasmine -- --tags=flaky --skip-tags=noCI
npm run test-bundle || EXIT_STATE=$?
exit $EXIT_STATE
;;

jasmine2)
retry npm run test-jasmine -- --tags=gl --skip-tags=noCI,flaky
retry npm run test-jasmine -- --tags=flaky --skip-tags=noCI
SHARDS=($(node $ROOT/tasks/shard_jasmine_tests.js --tag=gl))

for s in ${SHARDS[@]}; do
retry npm run test-jasmine -- "$s" --tags=gl --skip-tags=noCI
done

exit $EXIT_STATE
;;

Expand Down
6 changes: 6 additions & 0 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -136,6 +136,7 @@
"karma": "^3.0.0",
"karma-browserify": "^5.3.0",
"karma-chrome-launcher": "^2.0.0",
"karma-fail-fast-reporter": "^1.0.5",
"karma-firefox-launcher": "^1.0.1",
"karma-jasmine": "^1.1.2",
"karma-jasmine-spec-tags": "^1.0.1",
Expand Down
94 changes: 94 additions & 0 deletions tasks/shard_jasmine_tests.js
View file
Open in desktop
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,94 @@
var fs = require('fs');
var path = require('path');

var falafel = require('falafel');
var glob = require('glob');
var minimist = require('minimist');

var pathToJasmineTests = require('./util/constants').pathToJasmineTests;
var isJasmineTestIt = require('./util/common').isJasmineTestIt;

var argv = minimist(process.argv.slice(2), {
string: ['tag', 'limit'],
alias: {
tag: ['t'],
limit: ['l'],
},
default: {
limit: 20
}
});

var tag = argv.tag;
var limit = argv.limit;

glob(path.join(pathToJasmineTests, '*.js'), function(err, files) {
if(err) throw err;

var file2cnt = {};

@etpinardetpinardAug 21, 2018

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to note, with --limit=20 and --tag=gl, this fills up to:

{ 'cartesian_test.js': 1,
'cone_test.js': 5,
'gl_plot_interact_basic_test.js': 2,
'gl2d_click_test.js': 20,
'gl2d_date_axis_render_test.js': 4,
'gl2d_plot_interact_test.js': 28,
'gl2d_pointcloud_test.js': 2,
'gl2d_scatterplot_contour_test.js': 2,
'gl3d_plot_interact_test.js': 26,
'parcoords_test.js': 30,
'splom_test.js': 10,
'streamtube_test.js': 10 }


files.forEach(function(file) {
var code = fs.readFileSync(file, 'utf-8');
var bn = path.basename(file);

falafel(code, function(node) {
if(isJasmineTestIt(node, tag)) {
if(file2cnt[bn]) {
file2cnt[bn]++;
} else {
file2cnt[bn] = 1;
}
}
});
});

var ranking = Object.keys(file2cnt);
var runs = [];

// if 'it' count in file greater than threshold,
// run only this file separately,
// don't try to shard within file
Object.keys(file2cnt).forEach(function(f) {
if(file2cnt[f] > limit) {
runs.push(f);
ranking.splice(ranking.indexOf(f), 1);
}
});

// sort ranking in decreasing order
ranking.sort(function(a, b) { return file2cnt[b] - file2cnt[a]; });

var runi;
var cnt;

function newRun() {
var r0 = ranking[0];
runi = [r0];
cnt = file2cnt[r0];
ranking.shift();
}

function concat() {
runs.push(runi.join(','));
}

// try to match files with many tests with files not-that-many,
// by matching first rank with one or multiple trailing ranks.
newRun();
while(ranking.length) {
var rn = ranking[ranking.length - 1];

if((cnt + file2cnt[rn]) > limit) {
concat();
newRun();
} else {
runi.push(rn);
cnt += file2cnt[rn];
ranking.pop();
}
}
concat();

// print result to stdout
console.log(runs.join('\n'));
});
47 changes: 43 additions & 4 deletions tasks/test_syntax.js
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -8,6 +8,11 @@ var readLastLines = require('read-last-lines');
var eslint = require('eslint');
var trueCasePath = require('true-case-path');

var common = require('./util/common');
var isJasmineTestIt = common.isJasmineTestIt;
var isJasmineTestDescribe = common.isJasmineTestDescribe;
var hasJasmineTestTag = common.hasJasmineTestTag;

var constants = require('./util/constants');
var srcGlob = path.join(constants.pathToSrc, '**/*.js');
var libGlob = path.join(constants.pathToLib, '**/*.js');
Expand All@@ -28,26 +33,60 @@ assertES5();
// check for for focus and exclude jasmine blocks
function assertJasmineSuites() {
var BLACK_LIST = ['fdescribe', 'fit', 'xdescribe', 'xit'];
var TAGS = ['noCI', 'noCIdep', 'gl', 'flaky'];
var IT_ONLY_TAGS = ['gl', 'flaky'];
var logs = [];

var addTagPrefix = function(t) { return '@' + t; };

glob(combineGlobs([testGlob, bundleTestGlob]), function(err, files) {
files.forEach(function(file) {
var code = fs.readFileSync(file, 'utf-8');
var bn = path.basename(file);

falafel(code, {locations: true}, function(node) {
var lineInfo = '[line ' + node.loc.start.line + '] :';

if(node.type === 'Identifier' && BLACK_LIST.indexOf(node.name) !== -1) {
logs.push([
path.basename(file),
'[line ' + node.loc.start.line + '] :',
bn, lineInfo,
'contains either a *fdescribe*, *fit*,',
'*xdescribe* or *xit* block.'
].join(' '));
}
});

if(isJasmineTestIt(node)) {
if(hasJasmineTestTag(node)) {
if(TAGS.every(function(t) { return !hasJasmineTestTag(node, t); })) {
logs.push([
bn, lineInfo,
'contains an unrecognized tag,',
'not one of: ' + TAGS.map(addTagPrefix).join(', ')
].join(' '));
}
}

if(hasJasmineTestTag(node, 'gl') && hasJasmineTestTag(node, 'flaky')) {
logs.push([
bn, lineInfo,
'contains a @gl tag AND a @flaky tag, which is not allowed'
].join(' '));
}
}

IT_ONLY_TAGS.forEach(function(t) {
if(isJasmineTestDescribe(node, t)) {
logs.push([
bn, lineInfo,
'contains a', addTagPrefix(t), 'tag is a *describe* block,',
addTagPrefix(t), 'tags are only allowed in jasmine *it* blocks.'
].join(' '));
}
});
});
});

log('no jasmine suites focus/exclude blocks', logs);
log('no jasmine suites focus/exclude blocks or wrong tag patterns', logs);
});
}

Expand Down
27 changes: 27 additions & 0 deletions tasks/util/common.js
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -105,3 +105,30 @@ exports.formatEnumeration = function(list) {
return '`' + l + '`' + ending;
}).join(' ');
};

exports.hasJasmineTestTag = function(node, tag) {
var re = tag ?
new RegExp('@' + tag + '\\s') :
new RegExp('@' + '\\w');
return re.test(node.source());
};

function isJasmineBase(block, node, tag) {
return (
node.type === 'Literal' &&
node.parent &&
node.parent.type === 'CallExpression' &&
node.parent.callee &&
node.parent.callee.type === 'Identifier' &&
node.parent.callee.name === block &&
(tag === undefined || exports.hasJasmineTestTag(node, tag))
);
}

exports.isJasmineTestIt = function(node, tag) {
return isJasmineBase('it', node, tag);
};

exports.isJasmineTestDescribe = function(node, tag) {
return isJasmineBase('describe', node, tag);
};
42 changes: 19 additions & 23 deletions test/jasmine/karma.conf.js
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -7,16 +7,18 @@ var constants = require('../../tasks/util/constants');
var isCI = !!process.env.CIRCLECI;
var argv = minimist(process.argv.slice(4), {
string: ['bundleTest', 'width', 'height'],
'boolean': ['info', 'nowatch', 'verbose', 'Chrome', 'Firefox'],
'boolean': ['info', 'nowatch', 'failFast', 'verbose', 'Chrome', 'Firefox'],
alias: {
'Chrome': 'chrome',
'Firefox': ['firefox', 'FF'],
'bundleTest': ['bundletest', 'bundle_test'],
'nowatch': 'no-watch'
'nowatch': 'no-watch',
'failFast': 'fail-fast'
},
'default': {
info: false,
nowatch: isCI,
failFast: false,
verbose: false,
width: '1035',
height: '617'
Expand DownExpand Up@@ -53,6 +55,7 @@ if(argv.info) {
' - `--Chrome` (alias `--chrome`): run test in (our custom) Chrome browser',
' - `--Firefox` (alias `--FF`, `--firefox`): run test in (our custom) Firefox browser',
' - `--nowatch (dflt: `false`, `true` on CI)`: run karma w/o `autoWatch` / multiple run mode',
' - `--failFast` (dflt: `false`): exit karma upon first test failure',
' - `--verbose` (dflt: `false`): show test result using verbose reporter',
' - `--tags`: run only test with given tags (using the `jasmine-spec-tags` framework)',
' - `--width`(dflt: 1035): set width of the browser window',
Expand DownExpand Up@@ -100,11 +103,14 @@ if(isFullSuite) {

var pathToShortcutPath = path.join(__dirname, '..', '..', 'tasks', 'util', 'shortcut_paths.js');
var pathToStrictD3 = path.join(__dirname, '..', '..', 'tasks', 'util', 'strict_d3.js');
var pathToMain = path.join(__dirname, '..', '..', 'lib', 'index.js');
var pathToJQuery = path.join(__dirname, 'assets', 'jquery-1.8.3.min.js');
var pathToIE9mock = path.join(__dirname, 'assets', 'ie9_mock.js');
var pathToCustomMatchers = path.join(__dirname, 'assets', 'custom_matchers.js');

var reporters = (isFullSuite && !argv.tags) ? ['dots', 'spec'] : ['progress'];
if(argv.failFast) reporters.push('fail-fast');
if(argv.verbose) reporters.push('verbose');

function func(config) {
// level of logging
// possible values: config.LOG_DISABLE || config.LOG_ERROR || config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
Expand All@@ -118,7 +124,7 @@ function func(config) {
//
// See https://github.com/karma-runner/karma/commit/89a7a1c#commitcomment-21009216
func.defaultConfig.browserConsoleLogOptions = {
level: 'log'
level: 'debug'
};

config.set(func.defaultConfig);
Expand DownExpand Up@@ -154,7 +160,7 @@ func.defaultConfig = {
// See note in CONTRIBUTING.md about more verbose reporting via karma-verbose-reporter:
// https://www.npmjs.com/package/karma-verbose-reporter ('verbose')
//
reporters: (isFullSuite && !argv.tags) ? ['dots', 'spec'] : ['progress'],
reporters: reporters,

// web server port
port: 9876,
Expand DownExpand Up@@ -235,16 +241,18 @@ func.defaultConfig = {
suppressPassed: true,
suppressSkipped: false,
showSpecTiming: false,
// use 'karma-fail-fast-reporter' to fail fast w/o conflicting
// with other karma plugins
failFast: false

@alexcjohnsonalexcjohnsonAug 21, 2018

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this failFast shouldn't ever be used?

@etpinardetpinardAug 21, 2018

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Setting it to true won't break anything, but it won't actually fail fast.

@etpinardetpinardAug 21, 2018

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More on this topic: to make karma-spec-reporterfail fast, one needs to add it to the plugins list: https://github.com/mlex/karma-spec-reporter#configuration

When doing so, it conflicts with karma-browserify.

There might be a way, to list them (order dependent?) so that they don't conflict, but using karma-fail-fast-reporter was easy enough.

}
},

// e.g. when a test file does not container a given spec tags
failOnEmptyTestSuite: false
};

func.defaultConfig.preprocessors[pathToCustomMatchers] = ['browserify'];

if(isFullSuite) {
func.defaultConfig.files.push(pathToJQuery);
func.defaultConfig.preprocessors[testFileGlob] = ['browserify'];
} else if(isBundleTest) {
if(isBundleTest) {
switch(basename(testFileGlob)) {
case 'requirejs':
// browserified custom_matchers doesn't work with this route
Expand All@@ -271,14 +279,7 @@ if(isFullSuite) {
break;
}
} else {
// Add lib/index.js to non-full-suite runs,
// to make sure the registry is set-up correctly.
func.defaultConfig.files.push(
pathToJQuery,
pathToMain
);

func.defaultConfig.preprocessors[pathToMain] = ['browserify'];
func.defaultConfig.files.push(pathToJQuery);
func.defaultConfig.preprocessors[testFileGlob] = ['browserify'];
}

Expand All@@ -291,9 +292,4 @@ if(argv.Chrome) browsers.push('_Chrome');
if(argv.Firefox) browsers.push('_Firefox');
if(browsers.length === 0) browsers.push('_Chrome');

// add verbose reporter if specified
if(argv.verbose) {
func.defaultConfig.reporters.push('verbose');
}

module.exports = func;
Loading