◀ Course contents Part 4 · Module 4-06

Capstone: Your First Sprint

A real engineering job, in fifteen programs

No lesson in this module and no new syntax — you already have all of it. What you have not had is the job: a messy daily export, colleagues who describe what they want rather than which function to call, and an on-call engineer who will not schedule anything that cannot tell him whether it worked. Fourteen tickets land across the sprint. Friday, the tool ships.

The situation

You have joined Kestrel Supply Co

Read this once before you start. It is the whole of the handover you are going to get, which is also true of the real thing.

Kestrel Supply Co

A distributor with three sales regions. Every morning the billing system drops a CSV export, and every morning a human being opens it in a spreadsheet, builds a pivot table, and pastes three numbers into Slack. It takes twenty minutes and it has been wrong twice.

Engineer — first sprint

Your first piece of work is the thing everybody has wanted for a year and nobody has had time to build: report, a command-line tool that does the morning job in one command.

In cron by Friday

The tickets arrive in the order they were sent. The last one is the tool itself, reviewed by your lead and scheduled by your SRE.

Who is asking

Four people, and none of them will tell you which function to reach for. They will tell you what the tool has to do, and — more usefully — what went wrong the last time something did not. Those complaints are the requirements.

BC

Bea Cho · Finance analyst

Does the job by hand today. Knows every quirk in the export, because she has hit all of them.

YA

Yvonne Adeyemi · Head of RevOps

Posts the number every morning. Wants it ranked, wants it configurable, and reads the top two.

RM

Ravi Menon · SRE

Will schedule it. Cares about exit codes, streams, and error messages that name the file.

TL

Tomas Lindqvist · Engineering lead

Reviews the code. Cares that it can be imported, called and tested without a shell.

The export

One file, six rows, dropped by the billing system every morning. Every ticket that needs data writes this same file in its starter, so the numbers stay the same all week and you will recognise them by Wednesday.

sales.csv
regionquantityprice
eu31250
us4300
eu2500
jp1900
us6500
eux100

Revenue is quantity × price, so the five usable rows come to 9850 — eu 4750, us 4200, jp 900. The last row is not a mistake in the file: billing writes x while an order is mid-correction, it will never be fixed, and it is the row every naive version of this tool crashes on.

Handover notes

A CSV is all text. csv.DictReader hands you "3", not 3. Adding those together concatenates them rather than failing, which is the worst of both outcomes.

A bad row is not an emergency. Crash on it and one mid-correction stops the morning. Skip it silently and the totals quietly shrink. Skip it, count it, and report the count.

Nothing here reads your output. Cron reads an exit code; a pipe reads stdout. A tool that prints "failed" and exits 0 is indistinguishable from one that worked.

The file it writes is shared. The ranking lands where the dashboard reads it, so "do everything except the writing" has to be real — checked before the file is opened, not after.

The habit worth carrying into every ticket

Write functions that take what they need and hand back what they produced. A function that reads a file, prints a table and exits the process can only be tested by running the whole program and reading the screen. One that takes rows and returns pairs can be called with three made-up rows and checked in a line — which is exactly what the checks below do, and exactly what Tomas means every time he says "make it testable".

0 of 14 completed

Real Python runs right here in your browser — nothing to install, nothing sent to a server. The interpreter downloads once the first time you press Run, then stays cached.

01

To do

Bea Cho · Finance analyst

"Welcome aboard. So you know what you're replacing: every morning I download sales.csv from billing, open it in a spreadsheet, and the first thing I do is check the row count against yesterday's, because twice now the export has come through truncated and nobody noticed until the numbers were already in Slack."

Start where Bea starts. The starter writes today's export to disk for you — the same six rows every ticket this week uses — and your job is to read it back and say how big it is.

A CSV's first line is the header, so the number of records is one fewer than the number of lines. Work both out from the file rather than typing them: an export that arrives truncated is exactly the case where a hardcoded number lies.

Deliver: two lines.

7 lines
6 data rows
your_code.py
Python
Hint

with open("sales.csv") as f: lines = f.read().splitlines(). Then print(f"{len(lines)} lines") and print(f"{len(lines) - 1} data rows").

Output

      
    02

    To do

    Bea Cho · Finance analyst

    "Row count's a start, but I need the columns. Can you print each row out so I can eyeball it? Region, quantity, price, one row per line."

    csv.DictReader reads the header for you and hands back one dictionary per row, keyed by column name. Splitting on commas yourself works right up until a field contains one.

    Everything a CSV gives you is text

    row["quantity"] is the string "3", not the number 3. Adding those up concatenates them. Converting is your job, and tomorrow's ticket is about what happens when a value will not convert.

    Deliver: the header line as it appears in the file, then one line per row.

    region,quantity,price
    eu 3 1250
    us 4 300
    eu 2 500
    jp 1 900
    us 6 500
    eu x 100
    your_code.py
    Python
    Hint

    reader.fieldnames is the list of column names — ",".join(reader.fieldnames) gives the header line back. Then for row in reader: print(row["region"], row["quantity"], row["price"]).

    Output
    
          
      03

      To do

      Bea Cho · Finance analyst

      "You've seen the x in the quantity column. Billing does that occasionally when an order is mid-correction — it's not an error on our side and it'll never be fixed. Whatever you build has to skip those and keep going, and it has to tell me how many it skipped, because if that number jumps I want to know."

      Two decisions live in this ticket, and the second is the one that matters. Crashing on a bad row means one mid-correction stops the whole morning. Skipping it silently means the totals quietly shrink and nobody finds out. Skip it, count it, and say so.

      int("x") raises ValueError. Catch that specifically — a bare except would also swallow the typo in your own code.

      Deliver: the usable rows converted to numbers, and the two counts.

      loaded 5
      rejected 1
      your_code.py
      Python
      Hint

      Inside the loop, a try block doing row["quantity"] = int(row["quantity"]) and the same for price, then good.append(row). In `except ValueError:` do rejected += 1 and continue.

      Output
      
            
        04

        To do

        Yvonne Adeyemi · Head of RevOps

        "The number I actually post every morning is revenue by region. Quantity times price, added up per region. Right now Bea does it with a pivot table and I take her word for it."

        Accumulate into a dictionary keyed by region. defaultdict(int) saves the "have I seen this region before" branch: a key that has never been touched starts at 0, so += works the first time.

        Print them sorted by region name, so two runs of the same export produce the same output — a report whose row order moves around is a report people stop trusting.

        Deliver: one line per region, alphabetical.

        eu 4750
        jp 900
        us 4200
        your_code.py
        Python
        Hint

        for row in good: summed[row["region"]] += row["quantity"] * row["price"]. Then for region in sorted(summed): print(region, summed[region]).

        Output
        
              
          05

          To do

          Yvonne Adeyemi · Head of RevOps

          "Alphabetical is no good for the standup — I want biggest first, and honestly I only read the top two. Can you make the count a setting though? Some mornings I want all of them."

          Put the ranking in a function of its own. rank takes the totals and returns pairs, largest first — no printing, no files. That separation is what lets a test call it with rows it invented, and it is the shape the graded assignment is built around.

          sorted takes a key saying what to sort on and reverse=True for descending. Write the key as a named function rather than a lambda if it makes the intent clearer — either is fine, and one of them has a name.

          Deliver: rank(summed) returning (region, total) pairs largest first, then the top two printed.

          eu 4750
          us 4200
          your_code.py
          Python
          Hint

          def by_total(pair): return pair[1] — then return sorted(totals.items(), key=by_total, reverse=True). A dictionary's .items() gives exactly the pairs you want to sort.

          Output
          
                
            06

            To do

            Tomas Lindqvist · Engineering lead

            "Nice work on the totals. Now stop editing the file to change what it reads — the day someone changes the wrong line at 8am, we ship yesterday's numbers. It takes the path as an argument, and it takes the region filter as an option. That's the difference between a script and a tool."

            A script edited before each run is not a tool. The first thing to move out is whatever you keep changing at the top of the file.

            parser = argparse.ArgumentParser(prog="report")
            parser.add_argument("path")
            args = parser.parse_args(argv)

            A positional argument is required and unnamed — the thing the command is about. parse_args takes a list, which is what lets you test it without a shell.

            Your task: build a parser taking a path positional and a --region option defaulting to all, then print the two attributes for two different argument lists:

            sales.csv all
            sales.csv eu
            your_code.py
            Python
            Hint

            parser.add_argument("path") for the positional, and parser.add_argument("--region", default="all") for the option. The names on the parsed object come from the argument names.

            Output
            
                  
              07

              To do

              Yvonne Adeyemi · Head of RevOps

              "Two things. I want to choose how many rows I see, and I want to pick the output format — csv or json, nothing else. And if I typo one of those I want to be told straight away, not get a stack trace three seconds in."

              type=int converts and rejects. choices restricts and lists the valid values in the error. Both happen in the parser, so the rest of the program can assume the arguments are sane.

              parser.add_argument("--top", type=int, default=3)
              parser.add_argument("--format", choices=["csv", "json"], default="csv")

              A bad value makes argparse print the usage and exit with code 2 — the conventional code for "you called this wrongly".

              Your task: add both, then print the defaults, a converted value with its type, and the exit code from a bad one:

              3 csv
              5 int
              json
              exit 2
              your_code.py
              Python
              Hint

              add_argument("--top", type=int, default=3) and add_argument("--format", choices=["csv", "json"], default="csv"). Nothing else changes — argparse does the converting and the rejecting.

              Output
              
                    
                08

                To do

                Ravi Menon · SRE

                "Before this goes anywhere near cron it needs a switch that makes it do everything except the writing, so I can test the schedule without clobbering the shared file. And a verbose switch for when it misbehaves at 6am."

                action="store_true" makes a flag: False by default, True when the option appears. No value follows it.

                Watch the name. --dry-run arrives as args.dry_run, because a hyphen cannot be part of an attribute name.

                Your task: add --dry-run and --verbose, then print both for three argument lists:

                False False
                True False
                True True
                your_code.py
                Python
                Hint

                add_argument("--dry-run", action="store_true") and the same for --verbose. The attribute for the first one is dry_run, with an underscore.

                Output
                
                      
                  09

                  To do

                  Tomas Lindqvist · Engineering lead

                  "One structural note and then I'll leave you alone. Put the work in a function that takes the arguments and hands back an exit code. If the only way to run it is from a shell, the only way to test it is from a shell, and nobody will."

                  The shape that makes a tool testable is one function taking the arguments and returning the exit code.

                  def main(argv=None):
                      args = parser.parse_args(argv)
                      ...
                      return 0

                  Taking argv is what lets a check call main(["sales.csv", "--top", "2"]) without a shell, and passing None makes argparse fall back to the real command line — so the normal path is unaffected.

                  Your task: write main(argv=None) printing a summary line and returning 0. Call it twice:

                  reading sales.csv, top 3
                  0
                  reading other.csv, top 5
                  0
                  your_code.py
                  Python
                  Hint

                  args = parser.parse_args(argv), then print(f"reading {args.path}, top {args.top}"), then return 0. The print of the return value happens at the call site, which is why each call produces two lines.

                  Output
                  
                        
                    10

                    To do

                    Ravi Menon · SRE

                    "Cron doesn't read your output. It reads one number. If the export comes through empty and this thing exits 0, the pipeline marks it green and we find out on Monday. Give me the conventional codes."

                    The convention is worth following, because other programs depend on it:

                    0

                    Success

                    It did what it was asked.

                    1

                    Failure

                    It ran, and something was wrong — a missing file, no usable rows.

                    2

                    Usage error

                    You called it wrongly. argparse already does this one for you.

                    A shell script with set -e, a cron job and a CI step all check the code and none of them read the message. Without one, a failure is indistinguishable from a success.

                    Your task: write main(argv) returning 0 when there are rows to report and 1 when the file is empty. Print a line either way:

                    3 rows
                    0
                    no rows to report
                    1
                    your_code.py
                    Python
                    Hint

                    A guard: if not rows, print the complaint and return 1. Otherwise print the count and return 0. Early return keeps both paths at one level of indentation.

                    Output
                    
                          
                      11

                      To do

                      Ravi Menon · SRE

                      "I want to pipe this straight into the alerting job. That only works if the output is the answer and nothing else — no counts, no progress notes, no 'loaded 5 rows'. Send that stuff where it belongs."

                      Results go to stdout; diagnostics go to stderr. The reason is piping: report sales.csv | mail -s totals team should send the totals, not "rejected 2 rows".

                      print(row)
                      print(f"rejected {n} rows", file=sys.stderr)

                      Both reach a terminal, so a person running it still sees everything. The difference only shows when the output is redirected — which is exactly when it matters.

                      Your task: print the two data rows to stdout and the two diagnostics to stderr. Only the data will appear in the output below, which is the point:

                      eu 4750
                      us 1200
                      your_code.py
                      Python
                      Hint

                      A loop printing f"{region} {total}" for the rows, then two prints with file=sys.stderr for the diagnostics.

                      Output
                      
                            
                        12

                        To do

                        Ravi Menon · SRE

                        "Billing missed an upload last month and the tool we had said 'error'. That was the whole message. I spent forty minutes finding out which file it meant. Whatever this prints when it can't read something, it says the path."

                        FileNotFoundError already carries the path. Catching it and printing something vaguer is a downgrade — name the file, say what you were doing, put it on stderr, and return a failing code.

                        Your task: write main(argv) that reads the file and reports its line count, or reports the missing path on stderr and returns 1:

                        2 lines
                        0
                        1

                        The missing run prints nothing to stdout — its complaint went to stderr — so only its exit code appears.

                        your_code.py
                        Python
                        Hint

                        Wrap the open in try. In `except FileNotFoundError as error:` print f"cannot read {args.path}: {error.strerror}" to stderr and return 1. On success print the line count to stdout and return 0.

                        Output
                        
                              
                          13

                          To do

                          Yvonne Adeyemi · Head of RevOps

                          "The ranking file it writes is the one the dashboard reads. So the dry run has to be a genuine dry run — I want to see what it would have written, with the file untouched. If I can't trust that, I'll never run it on a Monday."

                          A dry run reads, validates, and reports what it would do — then stops before changing anything. On a tool that overwrites a file it is the difference between one people will run and one they will not.

                          if args.dry_run:
                              print(f"would write {len(rows)} rows to {args.out}")
                          else:
                              write(args.out, rows)

                          Your task: write main(argv) honouring --dry-run. The real run writes the file; the dry run says what it would have written and leaves the file alone:

                          would write 2 rows to out.txt
                          False
                          wrote 2 rows to out.txt
                          True

                          The booleans are whether the file exists after each run.

                          your_code.py
                          Python
                          Hint

                          An if on args.dry_run: print the "would write" line and return 0 before touching anything. Otherwise open args.out with "w", write the rows one per line, and print the "wrote" line.

                          Output
                          
                                
                            14

                            To do

                            Tomas Lindqvist · Engineering lead

                            "Last thing before you ship. I want to import your module from the nightly job and call main() directly — which I can't do if importing it runs the whole report and then kills my process. The exit goes at the very edge, not inside."

                            sys.exit belongs in the __main__ guard, not inside main. A function that exits the process cannot be called from a test, or reused by another tool.

                            if __name__ == "__main__":
                                sys.exit(main())

                            The guard also means importing the file runs nothing — the point Modules & the Standard Library made, and the reason one file can be both a tool and a library.

                            Your task: move the sys.exit out of main and into a guard at the bottom:

                            2
                            0
                            still running
                            2

                            The lab runs your code as the main module, so the guard fires and calls main a second time — which is the fourth line, and a useful thing to watch happen. Its sys.exit(0) then ends the program, which is why nothing follows it.

                            your_code.py
                            Python
                            Hint

                            Change sys.exit(0) to return 0, then add `if __name__ == "__main__":` with sys.exit(main()) indented under it at the bottom of the file.

                            Output
                            
                                  

                              Fri 11:00 — ship the tool

                              To do

                              Tomas Lindqvist · Engineering lead

                              "Everything's in the tickets. Put it in one file and I'll review it — and then Ravi puts it in cron on Monday, so it had better behave when nobody is watching it. Bea stops doing this by hand on Tuesday."

                              The week, assembled. Build report: a command-line tool that reads the sales export, totals it by region, prints the top few, and optionally writes the whole ranking to a file. Every piece of it is a ticket you have already done — the work is putting them in one file without any of them breaking the others.

                              The interface. Build a parser named report with:

                              • path — positional, the CSV to read
                              • --top — a whole number, default 2
                              • --out — a path to write the full ranking to, default None
                              • --dry-run — a flag

                              Write totals(rows): takes a list of DictReader dictionaries and returns (region, total) pairs, largest first. It must not open a file and must not print.

                              Write main(argv=None), which:

                              • reads the path with csv.DictReader, skipping rows whose numbers will not convert
                              • prints the top --top rows to stdout, as region total
                              • prints loaded N, rejected M to stderr
                              • with --out and no --dry-run, writes the full ranking there and prints wrote N rows to <path> to stdout
                              • with --out and --dry-run, prints would write N rows to <path> instead, and creates nothing
                              • returns 0 normally, 1 when the file is missing or no rows survived — printing the reason to stderr in both cases

                              Finish with the __main__ guard calling sys.exit(main()).

                              Print exactly eight lines — three runs, each followed by its exit code:

                              eu 4750
                              us 4200
                              0
                              eu 4750
                              us 4200
                              would write 3 rows to out.csv
                              0
                              1

                              The runs are: a plain run showing the top two; the same run with --out and --dry-run, which still shows the top two and then says what it would have written; and a run against a file that does not exist, which prints nothing to stdout because its complaint went to stderr.

                              your_code.py
                              Python
                              Hint

                              The parser takes "path", "--top" with type=int and default 2, "--out" with default None, and "--dry-run" with action="store_true". totals accumulates into a defaultdict(int) and returns sorted(..., key=..., reverse=True) with a named key function. In main: wrap the open in try/except FileNotFoundError, printing to stderr and returning 1. Loop the DictReader with a try round the two int() calls, counting rejects. If nothing survived, complain to stderr and return 1. Print the top rows with a slice, then the loaded/rejected line to stderr. Then, only when --out was given, either the "would write" line or the real write followed by "wrote". Return 0.

                              Output
                              
                                    

                                Notification