Enabling stdinput in the style of FastQC

[kerrycobb]

A feature of FastQC that is missing from falco is the ability to use stdin as input. I made a few changes to allow it to be done in the style of FastQC like cat <fastq files> | falco stdin:<prefix>.

I haven't tested it extensively but at a glance it looks like it's working with fastqc files. It assumes fastq format. I'm not sure how FastQC accomodates other formats as stdin. Perhaps with the --format flag.

I don't have much time right now to keep working on this. My fork is doing what I needed. Maybe this can inspire or help you to implement this feature. If you are receptive, I may be able to put some more effort in at some point to make this more robust and add some tests.

[andrewdavidsmith]

@kerrycobb Thanks for this. I appreciate the help. I won't be able to get to this for at least a week. Is there any chance you can update the test/falco.test script to add something that would cover this new kind of input? I'm not sure if you ran the make check after building with the autotools build. On my end I'll need time to run all of the things in the benchmark dir (using run_all_falco_tests.sh only) with the master branch vs. your PR and check that all files are the same except dates and times. Regarding the falco.test script, I think it's designed to cover types of inputs, rather than user args settings, so maybe a separate test script would be warranted, which would then appear in the line starting with TEST in Makefile.am.

[andrewdavidsmith]

@kerrycobb I'm preparing to release Falco v2, and your solution won't work directly because falco v2 does some look-ahead to get a rough estimate of the amount of data to process. I just wrote some of the code to handle pipes, with approximate assumptions on things like max read length and file size, but it's starting to look like I might need to set aside time later for this and get this into v2.1. I hope you'll try falco v2 when it's released, even if it can't do what you need.

[kerrycobb]

A feature of FastQC that is missing from falco is the ability to use stdin as input. I made a few changes to allow it to be done in the style of FastQC like cat <fastq files> | falco stdin:<prefix>.

I haven't tested it extensively but at a glance it looks like it's working with fastqc files. It assumes fastq format. I'm not sure how FastQC accomodates other formats as stdin. Perhaps with the --format flag.

I don't have much time right now to keep working on this. My fork is doing what I needed. Maybe this can inspire or help you to implement this feature. If you are receptive, I may be able to put some more effort in at some point to make this more robust and add some tests.

[andrewdavidsmith]

@kerrycobb Thanks for this. I appreciate the help. I won't be able to get to this for at least a week. Is there any chance you can update the test/falco.test script to add something that would cover this new kind of input? I'm not sure if you ran the make check after building with the autotools build. On my end I'll need time to run all of the things in the benchmark dir (using run_all_falco_tests.sh only) with the master branch vs. your PR and check that all files are the same except dates and times. Regarding the falco.test script, I think it's designed to cover types of inputs, rather than user args settings, so maybe a separate test script would be warranted, which would then appear in the line starting with TEST in Makefile.am.

[andrewdavidsmith]

@kerrycobb I'm preparing to release Falco v2, and your solution won't work directly because falco v2 does some look-ahead to get a rough estimate of the amount of data to process. I just wrote some of the code to handle pipes, with approximate assumptions on things like max read length and file size, but it's starting to look like I might need to set aside time later for this and get this into v2.1. I hope you'll try falco v2 when it's released, even if it can't do what you need.

[kerrycobb]

A feature of FastQC that is missing from falco is the ability to use stdin as input. I made a few changes to allow it to be done in the style of FastQC like cat <fastq files> | falco stdin:<prefix>.

I haven't tested it extensively but at a glance it looks like it's working with fastqc files. It assumes fastq format. I'm not sure how FastQC accomodates other formats as stdin. Perhaps with the --format flag.

I don't have much time right now to keep working on this. My fork is doing what I needed. Maybe this can inspire or help you to implement this feature. If you are receptive, I may be able to put some more effort in at some point to make this more robust and add some tests.

[andrewdavidsmith]

@kerrycobb Thanks for this. I appreciate the help. I won't be able to get to this for at least a week. Is there any chance you can update the test/falco.test script to add something that would cover this new kind of input? I'm not sure if you ran the make check after building with the autotools build. On my end I'll need time to run all of the things in the benchmark dir (using run_all_falco_tests.sh only) with the master branch vs. your PR and check that all files are the same except dates and times. Regarding the falco.test script, I think it's designed to cover types of inputs, rather than user args settings, so maybe a separate test script would be warranted, which would then appear in the line starting with TEST in Makefile.am.

[andrewdavidsmith]

@kerrycobb I'm preparing to release Falco v2, and your solution won't work directly because falco v2 does some look-ahead to get a rough estimate of the amount of data to process. I just wrote some of the code to handle pipes, with approximate assumptions on things like max read length and file size, but it's starting to look like I might need to set aside time later for this and get this into v2.1. I hope you'll try falco v2 when it's released, even if it can't do what you need.

[kerrycobb]

A feature of FastQC that is missing from falco is the ability to use stdin as input. I made a few changes to allow it to be done in the style of FastQC like cat <fastq files> | falco stdin:<prefix>.

I haven't tested it extensively but at a glance it looks like it's working with fastqc files. It assumes fastq format. I'm not sure how FastQC accomodates other formats as stdin. Perhaps with the --format flag.

I don't have much time right now to keep working on this. My fork is doing what I needed. Maybe this can inspire or help you to implement this feature. If you are receptive, I may be able to put some more effort in at some point to make this more robust and add some tests.

[andrewdavidsmith]

@kerrycobb Thanks for this. I appreciate the help. I won't be able to get to this for at least a week. Is there any chance you can update the test/falco.test script to add something that would cover this new kind of input? I'm not sure if you ran the make check after building with the autotools build. On my end I'll need time to run all of the things in the benchmark dir (using run_all_falco_tests.sh only) with the master branch vs. your PR and check that all files are the same except dates and times. Regarding the falco.test script, I think it's designed to cover types of inputs, rather than user args settings, so maybe a separate test script would be warranted, which would then appear in the line starting with TEST in Makefile.am.

[andrewdavidsmith]

@kerrycobb I'm preparing to release Falco v2, and your solution won't work directly because falco v2 does some look-ahead to get a rough estimate of the amount of data to process. I just wrote some of the code to handle pipes, with approximate assumptions on things like max read length and file size, but it's starting to look like I might need to set aside time later for this and get this into v2.1. I hope you'll try falco v2 when it's released, even if it can't do what you need.

[kerrycobb]

A feature of FastQC that is missing from falco is the ability to use stdin as input. I made a few changes to allow it to be done in the style of FastQC like cat <fastq files> | falco stdin:<prefix>.

I haven't tested it extensively but at a glance it looks like it's working with fastqc files. It assumes fastq format. I'm not sure how FastQC accomodates other formats as stdin. Perhaps with the --format flag.

I don't have much time right now to keep working on this. My fork is doing what I needed. Maybe this can inspire or help you to implement this feature. If you are receptive, I may be able to put some more effort in at some point to make this more robust and add some tests.

[andrewdavidsmith]

@kerrycobb Thanks for this. I appreciate the help. I won't be able to get to this for at least a week. Is there any chance you can update the test/falco.test script to add something that would cover this new kind of input? I'm not sure if you ran the make check after building with the autotools build. On my end I'll need time to run all of the things in the benchmark dir (using run_all_falco_tests.sh only) with the master branch vs. your PR and check that all files are the same except dates and times. Regarding the falco.test script, I think it's designed to cover types of inputs, rather than user args settings, so maybe a separate test script would be warranted, which would then appear in the line starting with TEST in Makefile.am.

[andrewdavidsmith]

@kerrycobb I'm preparing to release Falco v2, and your solution won't work directly because falco v2 does some look-ahead to get a rough estimate of the amount of data to process. I just wrote some of the code to handle pipes, with approximate assumptions on things like max read length and file size, but it's starting to look like I might need to set aside time later for this and get this into v2.1. I hope you'll try falco v2 when it's released, even if it can't do what you need.

[kerrycobb]

A feature of FastQC that is missing from falco is the ability to use stdin as input. I made a few changes to allow it to be done in the style of FastQC like cat <fastq files> | falco stdin:<prefix>.

I haven't tested it extensively but at a glance it looks like it's working with fastqc files. It assumes fastq format. I'm not sure how FastQC accomodates other formats as stdin. Perhaps with the --format flag.

I don't have much time right now to keep working on this. My fork is doing what I needed. Maybe this can inspire or help you to implement this feature. If you are receptive, I may be able to put some more effort in at some point to make this more robust and add some tests.

[andrewdavidsmith]

@kerrycobb Thanks for this. I appreciate the help. I won't be able to get to this for at least a week. Is there any chance you can update the test/falco.test script to add something that would cover this new kind of input? I'm not sure if you ran the make check after building with the autotools build. On my end I'll need time to run all of the things in the benchmark dir (using run_all_falco_tests.sh only) with the master branch vs. your PR and check that all files are the same except dates and times. Regarding the falco.test script, I think it's designed to cover types of inputs, rather than user args settings, so maybe a separate test script would be warranted, which would then appear in the line starting with TEST in Makefile.am.

[andrewdavidsmith]

@kerrycobb I'm preparing to release Falco v2, and your solution won't work directly because falco v2 does some look-ahead to get a rough estimate of the amount of data to process. I just wrote some of the code to handle pipes, with approximate assumptions on things like max read length and file size, but it's starting to look like I might need to set aside time later for this and get this into v2.1. I hope you'll try falco v2 when it's released, even if it can't do what you need.

[kerrycobb]

A feature of FastQC that is missing from falco is the ability to use stdin as input. I made a few changes to allow it to be done in the style of FastQC like cat <fastq files> | falco stdin:<prefix>.

I haven't tested it extensively but at a glance it looks like it's working with fastqc files. It assumes fastq format. I'm not sure how FastQC accomodates other formats as stdin. Perhaps with the --format flag.

I don't have much time right now to keep working on this. My fork is doing what I needed. Maybe this can inspire or help you to implement this feature. If you are receptive, I may be able to put some more effort in at some point to make this more robust and add some tests.

[andrewdavidsmith]

@kerrycobb Thanks for this. I appreciate the help. I won't be able to get to this for at least a week. Is there any chance you can update the test/falco.test script to add something that would cover this new kind of input? I'm not sure if you ran the make check after building with the autotools build. On my end I'll need time to run all of the things in the benchmark dir (using run_all_falco_tests.sh only) with the master branch vs. your PR and check that all files are the same except dates and times. Regarding the falco.test script, I think it's designed to cover types of inputs, rather than user args settings, so maybe a separate test script would be warranted, which would then appear in the line starting with TEST in Makefile.am.

[andrewdavidsmith]

@kerrycobb I'm preparing to release Falco v2, and your solution won't work directly because falco v2 does some look-ahead to get a rough estimate of the amount of data to process. I just wrote some of the code to handle pipes, with approximate assumptions on things like max read length and file size, but it's starting to look like I might need to set aside time later for this and get this into v2.1. I hope you'll try falco v2 when it's released, even if it can't do what you need.

[kerrycobb]

A feature of FastQC that is missing from falco is the ability to use stdin as input. I made a few changes to allow it to be done in the style of FastQC like cat <fastq files> | falco stdin:<prefix>.

I haven't tested it extensively but at a glance it looks like it's working with fastqc files. It assumes fastq format. I'm not sure how FastQC accomodates other formats as stdin. Perhaps with the --format flag.

I don't have much time right now to keep working on this. My fork is doing what I needed. Maybe this can inspire or help you to implement this feature. If you are receptive, I may be able to put some more effort in at some point to make this more robust and add some tests.

[andrewdavidsmith]

@kerrycobb Thanks for this. I appreciate the help. I won't be able to get to this for at least a week. Is there any chance you can update the test/falco.test script to add something that would cover this new kind of input? I'm not sure if you ran the make check after building with the autotools build. On my end I'll need time to run all of the things in the benchmark dir (using run_all_falco_tests.sh only) with the master branch vs. your PR and check that all files are the same except dates and times. Regarding the falco.test script, I think it's designed to cover types of inputs, rather than user args settings, so maybe a separate test script would be warranted, which would then appear in the line starting with TEST in Makefile.am.

[andrewdavidsmith]

@kerrycobb I'm preparing to release Falco v2, and your solution won't work directly because falco v2 does some look-ahead to get a rough estimate of the amount of data to process. I just wrote some of the code to handle pipes, with approximate assumptions on things like max read length and file size, but it's starting to look like I might need to set aside time later for this and get this into v2.1. I hope you'll try falco v2 when it's released, even if it can't do what you need.