02 Project organization

Sketch a project by hand, then generate one from a template and understand every folder

Plan the folders of the LTEE project on paper, compare with published advice, generate the project with Cookiecutter, and practice file naming and relative paths.
  • Time: 80 minutes in class, together with the lesson 03 hands-on, or about 60 minutes alone.
  • Slides: Project organization
  • You need: lesson 00 done (Pixi installed), paper and a pen.
  • You will have: the ltee-project folder that every later lesson builds on.

Overview

Everyone who has worked on a project for a few months knows the folder full of final.csv, final_v2.csv and final_USE_THIS.csv, and the questions that come with it: which one is right, and what made it?

Why organize a project?

A project changes as it grows. Data arrive in batches, scripts are rewritten, results are regenerated for a reviewer, and the student who did half the work moves on. After a few months nobody remembers which file came from which, and the folder is the only record left. You cannot predict who will need to understand the project later: you next year, a labmate who takes it over, a reviewer, a reader of the paper. So organize from the first day, not at the end. Give each kind of file one place, keep raw data apart and unchanged, and make every result traceable to the data and code that made it. Programs need this too: the workflows and automatic checks in later lessons can only find files that are where they are expected.

Organization is not one of the five pillars of computational reproducibility (literate programming, version control, environment control, data sharing, documentation; Ziemann et al. 2023), but it is the ground they stand on: each pillar assumes that files are where everyone expects them.

What you do in this lesson

You first design a layout yourself, then compare it with published advice, then generate one from a template so you can see every choice the template makes for you.

The running example is the LTEE project: variant calling on three samples of E. coli from Richard Lenski’s Long-Term Evolution Experiment, taken at 5,000, 15,000 and 50,000 generations. You do not need to know any genomics: treat the analysis as a given pipeline. What matters is how the project is organized and whether someone else can rerun it.

Learning outcomes

After this lesson you can:

  1. Create a new analysis project from a template with one command and say what belongs in each folder.
  2. Separate raw data, derived data, code and results so that raw data is never modified and results can be deleted and rebuilt.
  3. Name files so they sort correctly and can be read by people and programs.
  4. Write scripts with paths relative to the project folder, and show why absolute paths break.

Each outcome is checked by a step below; the last one by the Break it exercise.

Prerequisites

Lesson 00: Pixi works (pixi --version). Basic shell: cd, ls, mkdir.

1. Sketch first (10 minutes, on paper)

Before you look at any template, design the project yourself. A template is someone else’s decisions, frozen: if you never made the decisions, you cannot defend them, or change them when your project needs something different. You will analyze sequencing data for three samples. The analysis needs:

  • six FASTQ files (two per sample), downloaded from figshare;
  • a reference genome, downloaded from NCBI;
  • a table describing the samples (generation, strain, file names);
  • a few scripts and, later, a Snakemake and a Nextflow pipeline;
  • the software environment (Pixi files);
  • outputs: quality-control reports, alignments, variant calls, plots;
  • a report that explains the results;
  • a README, a license and a citation file.

On paper, draw the folders you would create and put each item in one. Mark which files you would never edit by hand and which you could delete and rebuild.

NoteThink, pair, share (5 minutes)

Compare your sketch with a neighbor. Where did you disagree? Which file was hardest to place? Keep your sketch: you will compare it with the template in step 4.

2. What others recommend

Two short papers are the classic advice for organizing computational biology projects. Read the key points below; the full papers are in Further reading.

Noble (2009), “A quick guide to organizing computational biology projects”:

  • one folder per project, with the same layout every time;
  • separate folders for data, src (code), bin (compiled tools), results and doc;
  • raw data stays as it was received; everything else is made from it by code;
  • a dated lab notebook (results/2009-01-15/) records what you did and why;
  • a “driver” script reruns the whole analysis.

Wilson et al. (2017), “Good enough practices in scientific computing”:

  • put each project in its own directory, named after the project;
  • put text documents in doc/, raw data and metadata in data/, files generated during cleanup and analysis in results/, source code in src/;
  • name all files to reflect their content or function;
  • keep raw data read-only, and create the data you analyze with code.

Both agree on the main rule: raw data in, code in the middle, results out, and results can always be regenerated. Raw data is the only part of a project you cannot regenerate: keep it as you received it, and any mistake made later can be fixed by rerunning the code from it.

3. Generate the project from the template

A template creates the same, well-thought-out layout every time, so you do not have to remember it. We use Cookiecutter, run through Pixi, so you do not need to install it.

Create a folder for this course and go into it:

mkdir -p ~/cbio7501
cd ~/cbio7501

Run the template. Cookiecutter asks a few questions; press Enter to keep the value in brackets. Use ltee-project as the folder name:

pixi exec --spec cookiecutter --spec git -- \
  cookiecutter gh:khan-lab/cbio7501-f26 --directory templates/cookiecutter-repro-project
  [1/12] Project name (a few words) (LTEE variant calling):
  [2/12] Folder and repository name (lowercase, dashes) (ltee-variant-calling): ltee-project
  [3/12] One sentence: what does this project do? (A reproducible analysis project): Variant calling on LTEE E. coli samples
  [4/12] Your given name(s) (Your): Ada
  ...
TipSame thing, without the questions

You can give all answers on the command line instead, which is quicker when you make many projects. Use your own name and ORCID iD:

pixi exec --spec cookiecutter --spec git -- \
  cookiecutter --no-input gh:khan-lab/cbio7501-f26 --directory templates/cookiecutter-repro-project \
  project_name="LTEE variant calling" project_slug="ltee-project" \
  short_description="Variant calling on three E. coli samples from the Long-Term Evolution Experiment" \
  author_given_names="Ada" author_family_names="Lovelace" author_email="ada@example.org" \
  github_username="ada-lovelace" license="MIT"

Expected output ends with:

Created ltee-project/

Next steps:
  cd ltee-project
  pixi install            # create the environment and pixi.lock
  pixi run check          # check that the tools work

(Not now: you create the environment in lesson 05.)

4. Walk through the layout

List the folders:

cd ~/cbio7501/ltee-project
find . -type d | sort
.
./.github
./.github/workflows
./config
./containers
./data
./data/processed
./data/raw
./docs
./envs
./metadata
./notebooks
./reports
./results
./scripts
./workflow

Every folder has a README.md that says what goes in it. Read two of them:

cat data/raw/README.md
cat results/README.md

The main README.md has a table of all folders. Here is the logic in one picture:

  data/raw/  ──code──►  data/processed/  ──code──►  results/  ──►  reports/
  (read-only,           (derived,                   (figures,      (Quarto
   downloaded)           rebuildable)                tables)        report)
       ▲                                                  ▲
  metadata/  (what each file is, where it came from, checksums)
  scripts/, workflow/, config/  (the code and its parameters)
  pixi.toml + pixi.lock  (the software)

The picture is also a test of the layout: if you delete data/processed/ and results/, one command must bring them back (from lesson 05 on, pixi run reproduce). Anything that cannot be rebuilt that way is either raw data, which belongs in data/raw/ and is never changed, or a step someone did by hand, which belongs in a script.

NoteCompare with your sketch (5 minutes)

Go through your paper sketch. For each difference, decide: is the template better, is your version better, or is it a matter of taste? Two points worth discussing:

  • The template has no final/ folder and no dated results/2026-10-27/ folders (Noble’s lab notebook). Results are regenerated by code; history is kept by Git (lesson 04).
  • metadata/ is separate from data/: it is small, written by people and committed to Git, while data/ is large, downloaded and not committed.

What else the template created

ls -a
cat pixi.toml
  • pixi.toml: the software this project needs (Python, Quarto, Jupyter, pandas, matplotlib). You learn how it works in lesson 05.
  • .gitignore: tells Git (lesson 04) not to track raw data, results and the installed environment.
  • LICENSE and CITATION.cff: how others may reuse and cite the project (lesson 11).
  • .github/workflows/reproduce.yml: a check that reruns the project on a fresh computer every time you push (from lesson 04; it gets real work in lesson 05).

5. Name files so that they sort and explain themselves

File names are the first metadata anyone sees, and programs select files by pattern: data/raw/fastq/*_1.trim.sub.fastq, or a Snakemake wildcard in lesson 09, finds the right files only if every name follows the same rules. A good file name is readable by people, easy for programs to parse, and sorts in a useful order. Rules we follow in this course:

  • no spaces or special characters; use - or _;
  • dates as YYYY-MM-DD (ISO 8601), so they sort in time order;
  • numbers with leading zeros (s01, s02, …, s12);
  • the most important part first (the sample ID, if you look files up by sample);
  • the same parts in the same order in every name, with the abbreviations explained in metadata/README.md;
  • a version number only for things you release (v001); everything else is versioned by Git.

The LTEE files follow these rules already: SRR2584863_1.trim.sub.fastq is run SRR2584863, read 1 of the pair, trimmed, subsampled.

Try it yourself

  1. These names were sorted alphabetically by a computer. What problems do you see?

    12-10-2026 alignment SRR2589044.bam
2026-10-13_SRR2584866_aligned.bam
2026-10-9_SRR2584863_aligned.bam
Final results (use this).xlsx
SRR2584863_aligned_v2.bam
sample1.bam
sample10.bam
sample2.bam
    • Two date formats; 12-10-2026 is ambiguous (December or October?) and does not sort.
    • 2026-10-9 lacks a leading zero, so it sorts after 2026-10-13.
    • Spaces and brackets break shell commands unless quoted.
    • sample10 sorts between sample1 and sample2 (no leading zeros).
    • _v2 and “use this” are manual version control; use Git instead.
    • The sample ID moves around, so you cannot list all files of a sample with one pattern.

    A consistent version: SRR2584863.aligned.sorted.bam, SRR2584866.aligned.sorted.bam, SRR2589044.aligned.sorted.bam, made by code in results/bam/.

  2. Which name in each group follows the rules best?

    1. variant counts 5Aug2026.csv b) variant-counts_2026-08-05.csv c) counts.csv

    b): no spaces, ISO date, says what is inside. c) is too vague; a) has spaces and a date that does not sort.

6. Start the README now

A README written on day one is a plan; one written at the end is a chore. Open README.md in your editor and fill in the Data section. You can copy this:

## Data

Raw reads: a subset (175,000 read pairs per sample) of three *E. coli* clones
from the Long-Term Evolution Experiment (population Ara-3, generations 5,000,
15,000 and 50,000), prepared by Data Carpentry:
<https://doi.org/10.6084/m9.figshare.7726454> (file `sub.tar.gz`, CC0).
Reference genome: *E. coli* B str. REL606, GenBank CP000819.1 (NCBI).
Details, sample sheet and checksums are in `metadata/`.

You will download the data and write the sample sheet in the lesson 03 hands-on.

Break it: absolute paths

A script that only works in one folder on one laptop is not reproducible. See it fail. First write a tiny script that reads a file through an absolute path (the full path on your computer, which the shell fills in with $PWD):

cd ~/cbio7501/ltee-project
echo "print(open('$PWD/metadata/README.md').readline())" > scripts/first_line_absolute.py
cat scripts/first_line_absolute.py
pixi exec python scripts/first_line_absolute.py

It works. Now do what a colleague does when they download your project: put it in a different folder. Here we just rename the project folder:

cd ~/cbio7501
mv ltee-project ltee-project-copy
cd ltee-project-copy
pixi exec python scripts/first_line_absolute.py
FileNotFoundError: [Errno 2] No such file or directory: '/Users/ada/cbio7501/ltee-project/metadata/README.md'

The fix is a relative path, which works wherever the project folder is, as long as you run scripts from the project folder:

scripts/first_line.py
print(open("metadata/README.md").readline())
pixi exec python scripts/first_line.py

Clean up and rename the folder back:

rm scripts/first_line_absolute.py scripts/first_line.py
cd ~/cbio7501
mv ltee-project-copy ltee-project
cd ltee-project
Note

Paths like C:\Users\ada\Desktop\data.csv or /Users/ada/... in shared code are one of the most common reasons a project fails for someone else. In every later lesson, run commands from the project folder and use relative paths.

Troubleshooting

Problem What to do
pixi: command not found Do lesson 00 first, then open a new terminal.
git: command not found or “Unable to clone” Cookiecutter downloads the template with Git. Keep --spec git in the command, or install Git (lesson 00).
Error: "ltee-project" directory already exists You ran the template twice. Remove or rename the old folder, or answer with another folder name.
ERROR: 'My Project' is not a good folder name The folder name must be lowercase with dashes, e.g. ltee-project.
The questions look stuck Cookiecutter is waiting for your answer: type it and press Enter.
Windows: the files are not visible in Explorer You are working inside WSL2: run explorer.exe . in the project folder.

Apply it to your own project

From this lesson on you also work on your capstone: your own research project, separate from the LTEE project (capstone rubric). Choose an existing or ongoing analysis of yours, or one manageable part of it, such as one dataset and the analysis behind one result. Generate a project for it with the same template, from ~/cbio7501, with your own project name, and copy your existing scripts into scripts/ (your data comes in lesson 03). Then, in its README.md, write one paragraph: the question, the data (source, size, license) and what you expect to produce. This is your project proposal.

Key points

  • Decide the layout before the first file arrives; use the same layout in every project.
  • Raw data is read-only and downloaded by code; derived data and results are rebuilt by code.
  • A template saves time, but know why each folder is there.
  • Name files so they sort and can be parsed: no spaces, ISO dates, leading zeros, consistent order.
  • Use paths relative to the project folder; absolute paths break on any other computer.

Further reading