Skip to content

[lab] added cmu15-213 "data lab" - #62

Merged
tareknaser merged 2 commits into
mainfrom
15-213lab
Jan 12, 2026
Merged

[lab] added cmu15-213 "data lab"#62
tareknaser merged 2 commits into
mainfrom
15-213lab

Conversation

@jq-huang

@jq-huangjq-huang commented Jan 10, 2026

Copy link
Copy Markdown
Collaborator

Description

Added "Data Lab" from CMU 15-213 course, used the self-study handout, see at csapp lab assignments

Changes

  • Added the lab

Testing

Test the lab with qwen-plus model:

 "config": {
"model": "openai/qwen-plus",
"max_steps": 50,
"max_cost": 5.0,
"timestamp": "2026-01-10T03:33:24.372032"
}

It reached the limited steps and got the result:

Limit reached: Step limit reached: 50/50
Running evaluation...
Evaluation failed on attempt 1, retrying...
Retry attempt 2/3...
Evaluation failed on attempt 2, retrying...
Retry attempt 3/3...
Copying output files to outputs/run_20260110_033009/files/cmu_15_213_data_lab
✗ Completed in 195.0s (50 steps, $0.0000)
✗ limit_reached | $0.0000
======================================================================
Pass rate: 0/1 (0.0%)
Total cost: $0.0000
Results: outputs/run_20260110_033009
======================================================================
=== Evaluating Data Lab ===
Step 0: Verifying protected files integrity...
Protected files integrity verified
Step 1: Checking coding rule compliance with dlc...
dlc warnings/errors:
dlc:bits.c:167:isTmax: Illegal operator (<<)
dlc check passed
Step 2: Building btest...
btest compiled successfully
Step 3: Running btest...
btest output:
Score Rating Errors Function
1 1 0 bitXor
1 1 0 tmin
1 1 0 isTmax
2 2 0 allOddBits
2 2 0 negate
3 3 0 isAsciiDigit
3 3 0 conditional
3 3 0 isLessOrEqual
4 4 0 logicalNeg
0 4 1 howManyBits
0 4 1 floatScale2
0 4 1 floatFloat2Int
0 4 1 floatPower2
Total points: 20/36
FAIL: btest detected errors in some functions

Checklist

  • Tests pass locally
  • Code follows project style guidelines
  • Documentation updated (if needed)

CopilotAI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds the CMU 15-213 "Data Lab" programming assignment to the courselab_bench benchmark suite. The Data Lab is a well-known assignment from Carnegie Mellon's Introduction to Computer Systems course that challenges students to implement bit-level manipulation functions and floating-point operations with restricted operators.

Changes:

  • Added course registration for CMU 15-213 to courses.json
  • Added complete Data Lab assignment with task description, starter files, build system, and evaluation scripts
  • Configured Docker environment with necessary 32-bit compilation support and Perl dependencies

Reviewed changes

Copilot reviewed 18 out of 19 changed files in this pull request and generated 6 comments.

Show a summary per file
FileDescription
courses.jsonRegisters the CMU 15-213 course in the benchmark system
task.mdProvides detailed instructions and documentation for the Data Lab
starter_files/.c/.hSource files including student starter code, test harness, and utility programs
starter_files/dlcBinary compiler for checking coding rule compliance
starter_files/.pl/.pmPerl driver scripts for automated grading
starter_files/MakefileBuild system for compiling the test harness
preprocess.shSetup script that installs dependencies and validates file integrity
evaluate.shEvaluation script that runs dlc checks and btest for correctness
config.jsonConfiguration defining the lab environment and file mappings

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@tareknasertareknaser left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. It looks good to me

Signed-off-by: Tarek <tareknaser360@gmail.com>
@tareknaser
tareknaser merged commit 5c6b412 into mainJan 12, 2026
6 checks passed
@tareknaser
tareknaser deleted the 15-213lab branch January 15, 2026 16:56
@xuafengxuafeng mentioned this pull request Feb 2, 2026
tareknaser added a commit that referenced this pull request Feb 5, 2026
* added cmu15-213 data lab
* docs(courselab): add note about infrastructure restrictions
Signed-off-by: Tarek <tareknaser360@gmail.com>
---------
Signed-off-by: Tarek <tareknaser360@gmail.com>
Co-authored-by: Tarek <tareknaser360@gmail.com>
jq-huang added a commit that referenced this pull request Feb 8, 2026
* added cmu15-213 data lab
* docs(courselab): add note about infrastructure restrictions
Signed-off-by: Tarek <tareknaser360@gmail.com>
---------
Signed-off-by: Tarek <tareknaser360@gmail.com>
Co-authored-by: Tarek <tareknaser360@gmail.com>
jq-huang added a commit that referenced this pull request Feb 8, 2026
* added cmu15-213 data lab
* docs(courselab): add note about infrastructure restrictions
Signed-off-by: Tarek <tareknaser360@gmail.com>
---------
Signed-off-by: Tarek <tareknaser360@gmail.com>
Co-authored-by: Tarek <tareknaser360@gmail.com>
jq-huang added a commit that referenced this pull request Feb 8, 2026
* added cmu15-213 data lab
* docs(courselab): add note about infrastructure restrictions
Signed-off-by: Tarek <tareknaser360@gmail.com>
---------
Signed-off-by: Tarek <tareknaser360@gmail.com>
Co-authored-by: Tarek <tareknaser360@gmail.com>
jq-huang added a commit that referenced this pull request Feb 8, 2026
* added cmu15-213 data lab
* docs(courselab): add note about infrastructure restrictions
Signed-off-by: Tarek <tareknaser360@gmail.com>
---------
Signed-off-by: Tarek <tareknaser360@gmail.com>
Co-authored-by: Tarek <tareknaser360@gmail.com>
jq-huang added a commit that referenced this pull request Feb 8, 2026
* added cmu15-213 data lab
* docs(courselab): add note about infrastructure restrictions
Signed-off-by: Tarek <tareknaser360@gmail.com>
---------
Signed-off-by: Tarek <tareknaser360@gmail.com>
Co-authored-by: Tarek <tareknaser360@gmail.com>
Couen pushed a commit to Couen/system-intelligence-benchmark that referenced this pull request Feb 25, 2026
* added cmu15-213 data lab
* docs(courselab): add note about infrastructure restrictions
Signed-off-by: Tarek <tareknaser360@gmail.com>
---------
Signed-off-by: Tarek <tareknaser360@gmail.com>
Co-authored-by: Tarek <tareknaser360@gmail.com>
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

@jq-huang@tareknaser