Support for reference sequences longer than 65535

[boulund]

I was hoping to use lambda for mapping fairly short peptides to reference genome sequences, which I guess it isn't really intended for... Is there a reason for this, perhaps the alignment approach used in lambda isn't well suited for this type of mapping?

[h-2]

Well, I had to make the decision for the most common use cases, because using position types that are larger increases the size of the index file measurably (for every one else as well).
You can change this line to contain a 32bit integer:
https://github.com/seqan/lambda/blob/master/src/options.hpp#L57
and rebuild lambda, then it should "just work".

I can make a compile time option for this in the future so you could specify e.g. -DLAMBDA_LONG_PROTEIN_SUBJ_SEQS 😃

[boulund]

Hi,

I respect your decision of using 16 bit uints for the index, that's fine. I'll definitely try the change you suggested. Hopefully I'll get an environment up where I can compile this on my own (been using the precompiled binaries so far). I'll report back as soon as I have time to try this.

[boulund]

While we're at it. I noticed the expected number of sequences is stored as uint32. That also seems a bit low for what I normally use [1]. I would suggest adding options to compile with 64 bit integers in these fields as well, since capping it merely 4.3 million sequences isn't a lot. Without looking at the entire code base, I'm not sure if this would this be possible with lambda, are there any implications I'm not considering?

[1] Genbank's non-redundant protein database currently contains more than 57 million sequences...

[h-2]

I'll report back as soon as I have time to try this.

Ok, please let me know if it works!

While we're at it. I noticed the expected number of sequences is stored as uint32. That also seems a bit low for what I normally use [1]. I would suggest adding options to compile with 64 bit integers in these fields as well, since capping it merely 4.3 million sequences isn't a lot.

32bit means up to 4.3 billion sequences, I hope that's enough for the next two or three years 😄

[boulund]

Haha. Whoops. My bad 😆. Disregard that, I still haven't woken up today it seems.

[boulund]

I'm not having any luck trying to build lambda 😞. Trying to build from the lambda-next branch, but also tried building the master branch just to see if that works (it didn't). Tried it on three different machines:

• RHEL 7.2, 2x20 core Intel Xeon E5-2650 v3 with 192GB RAM, GCC 4.9.4 built from scratch (this morning), cmake 3.6.1.
• CentOS 6.7, 20 core Intel Xeon E5-2650 v3 with 132GB RAM, GCC 6.2.0, cmake 3.3.1.
• Debian stretch/sid (Bash on Ubuntu on Windows 10 insider preview), 4 core i7-6600U, 16GB RAM, GCC 5.4.0-6ubuntu1~16.04.2, cmake 3.5.1

Besides lots of errors like this:
/home/boulund/src/lambda.git/src/lambda.hpp:2067:36: error: ‘_prepareAndRunSimdAlignment’ was not declared in this scope, which I guess is related to the SIMD-development that's currently in progress?
I also see this (sorry for this one being in Swedish):

/usr/bin/ld: försökte länka det dynamiska objektet '/home/boulund/anaconda3/lib/libz.so' statiskt. 
collect2: error: ld returned 1 exit status 
src/CMakeFiles/lambda_indexer.dir/build.make:95: receptet för målet ”bin/lambda_indexer” misslyckades
make[2]: *** [bin/lambda_indexer] Fel 1 
CMakeFiles/Makefile2:124: receptet för målet ”src/CMakeFiles/lambda_indexer.dir/all” misslyckades 
make[1]: *** [src/CMakeFiles/lambda_indexer.dir/all] Fel 2 
make[1]: *** Inväntar oavslutade jobb... 
/usr/bin/ld: försökte länka det dynamiska objektet '/home/boulund/anaconda3/lib/libz.so' statiskt. 
collect2: error: ld returned 1 exit status 
src/CMakeFiles/lambda.dir/build.make:95: receptet för målet ”bin/lambda” misslyckades 
make[2]: *** [bin/lambda] Fel 1 
CMakeFiles/Makefile2:87: receptet för målet ”src/CMakeFiles/lambda.dir/all” misslyckades 
make[1]: *** [src/CMakeFiles/lambda.dir/all] Fel 2 
Makefile:149: receptet för målet ”all” misslyckades 
make: *** [all] Fel 2 

Based on these messages I was thinking maybe my anaconda3 installation is messing things up, but removing all anaconda-stuff from my $PATH didn't change anything.

On the second machine the build and linking of lambda_indexer seems to work, and it appears to run like it should.

[h-2]

Did you clone the master branch with the --recursive argument? Otherwise lambda will use SeqAn from your system which might be too old.

I'm not having any luck trying to build lambda . Trying to build from the lambda-next branch, but also tried building the master branch just to see if that works (it didn't).

Please try only with the master branch for now. And can you post a full log of the cmake-run? [for make the last parts are enough, but for cmake I need the full log]

I also see this (sorry for this one being in Swedish):

Inga problem, jag förstår lite ;)

[boulund]

Yes, I used the --recursive argument. I don't have the SeqAn library installed system-wide (I've never had much success with SeqAn to be honest; very happy you supply precompiled binaries).

Strangely enough I managed to complete a build from the master branch just now. The compiled binaries don't work, however:

[bin]$ ./lambda 
./lambda: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by ./lambda) 
[bin]$ ./lambda_indexer 
./lambda_indexer: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by ./lambda_indexer)

Here is ldd output for these binaries

[bin]$ ldd lambda 
./lambda: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by ./lambda) 
 linux-vdso.so.1 => (0x00007fffeaffe000) 
 librt.so.1 => /lib64/librt.so.1 (0x00007fb16ab66000) 
 libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb16a94a000) 
 libz.so.1 => /lib64/libz.so.1 (0x00007fb16a733000) 
 libbz2.so.1 => /lib64/libbz2.so.1 (0x00007fb16a523000) 
 libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fb16a21a000) 
 libm.so.6 => /lib64/libm.so.6 (0x00007fb169f17000) 
 libgomp.so.1 => /lib64/libgomp.so.1 (0x00007fb169d00000) 
 libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fb169aea000) 
 libc.so.6 => /lib64/libc.so.6 (0x00007fb169728000) 
 /lib64/ld-linux-x86-64.so.2 (0x00007fb16ad86000) 
[bin]$ ldd lambda_indexer 
./lambda_indexer: /lib64/libstdc++.so.6: version `CXXABI_1.3.8' not found (required by ./lambda_indexer)
 linux-vdso.so.1 => (0x00007ffe0df5f000) 
 librt.so.1 => /lib64/librt.so.1 (0x00007fa8d7c90000) 
 libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa8d7a74000) 
 libz.so.1 => /lib64/libz.so.1 (0x00007fa8d785d000) 
 libbz2.so.1 => /lib64/libbz2.so.1 (0x00007fa8d764d000) 
 libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fa8d7344000) 
 libm.so.6 => /lib64/libm.so.6 (0x00007fa8d7041000) 
 libgomp.so.1 => /lib64/libgomp.so.1 (0x00007fa8d6e2a000) 
 libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fa8d6c14000) 
 libc.so.6 => /lib64/libc.so.6 (0x00007fa8d6852000) 
 /lib64/ld-linux-x86-64.so.2 (0x00007fa8d7eb0000) 

Not sure what exactly you want from cmake, but here are two log files I could find. Hope they are of some use.
CMakeError.log.txt
CMakeOutput.log.txt

[boulund]

Deleted everything and started over again and tried another build on my laptop (Bash on Ubuntu on Windows thing) from the master branch, and it seems to succeed now. I could even produce static binaries that seem to run on my main crunching machine (the first of the three listed in my previous post). However, I got this output during compilation:

Scanning dependencies of target lambda 
Scanning dependencies of target lambda_indexer 
[ 50%] Building CXX object src/CMakeFiles/lambda_indexer.dir/lambda_indexer.cpp.o 
[ 50%] Building CXX object src/CMakeFiles/lambda.dir/lambda.cpp.o 
[ 75%] Linking CXX executable ../bin/lambda_indexer 
/usr/lib/gcc/x86_64-linux-gnu/5/libgomp.a(target.o): I funktionen "gomp_target_init": 
(.text+0xba): varning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
[ 75%] Built target lambda_indexer 
[100%] Linking CXX executable ../bin/lambda 
/usr/lib/gcc/x86_64-linux-gnu/5/libgomp.a(target.o): I funktionen "gomp_target_init": 
(.text+0xba): varning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
[100%] Built target lambda 

Not sure if this is a problem though. It seems to be working now.

[h-2]

It looks like you are using a non-system compiler, but then linking against system libraries, this can lead to all sorts of problems, unreleated to Lambda :)
This might already be solvable by removing your cmake files and then rerunning cmake with -DLAMBDA_STATIC_BUILD=ON.
For dynamic linking you need to pass the correct rpath to your compiler:

-DCMAKE_CXX_FLAGS="-L/path/to/lib" \
-DCMAKE_EXE_LINKER_FLAGS="-Wl,-rpath=/path/to/lib"

[However static builds might break on centos, because they don't provide the static lib files for zlib, this is the error you are seeing up at the top]

[h-2]

Not sure if this is a problem though. It seems to be working now.

Yes, these warnings can be ignored!

[boulund]

Yeah. Compiling stuff that requires newer GCC versions than what RedHat repositories have available is frustrating, to say the least... 😞

Despite my success with the static binary from my laptop, I'll try building static binaries on my main machine as well, just to see if I can get it to work.

The binaries complain about not being built in release mode, and they are indeed very slow. Can I do something about this?