Welcome

Hello, I'm Mahmoud, and these are my notes. Since you're reading something I've written, I want to share a bit about how I approach learning and what you can expect here.

Want to know more about me? Check out my blog at mahmoud.ninja

My Philosophy

I believe there are no shortcuts to success. To me, success means respecting your time, and before investing that time, you need a plan rooted in where you want to go in life.

Learn, learn, learn, and when you think you've learned enough, write it down and share it.

About These Notes

I don't strive for perfectionism. Sometimes I write something and hope someone will point out where I'm wrong so we can both learn from it. That's the beauty of sharing knowledge: it's a two-way street.

I tend to be pretty chill, and I occasionally throw in some sarcasm when it feels appropriate. These are my notes after all, so please don't be annoyed if you encounter something that doesn't resonate with you, just skip ahead.

I'm creating this resource purely out of love for sharing and teaching. Ironically, I'm learning more by organizing and explaining these concepts than I ever did just studying them. Sharing is learning. Imagine if scientists never shared their research, we'd still be in the dark ages.

Licensing & Copyright

Everything I create here is released under Creative Commons (CC BY 4.0). You're free to share, copy, remix, and build upon this material for any purpose, even commercially, as long as you give appropriate credit.

Important Legal Notice

I deeply respect intellectual property rights. I will never share copyrighted materials, proprietary resources, or content that was shared with our class under restricted access. All external resources linked here are publicly available or properly attributed.

If you notice any copyright violations or improperly shared materials, please contact me immediately at mahmoudahmedxyz@gmail.com, and I will remove the content right away and make necessary corrections.

Final Thoughts

I have tremendous respect for everyone in this learning journey. We're all here trying to understand complex topics, and we all learn differently. If these notes help you even a little bit, then this project has served its purpose.

Linux Fundamentals

The History of Linux

In 1880, the French government awarded the Volta Prize to Alexander Graham Bell. Instead of going to the Maldives (kidding...he had work to do), he went to America and opened Bell Labs.

This lab researched electronics and something revolutionary called the mathematical theory of communications. In the 1950s came the transistor revolution. Bell Labs scientists won 10 Nobel Prizes...not too shabby.

But around this time, Russia made the USA nervous by launching the first satellite, Sputnik, in 1957. This had nothing to do with operating systems, it was literally just a satellite beeping in space, but it scared America enough to kickstart the space race.

President Eisenhower responded by creating ARPA (Advanced Research Projects Agency) in 1958, and asked James Killian, MIT's president, to help develop computer technology. This led to Project MAC (Mathematics and Computation) at MIT.

Before Project MAC, using a computer meant bringing a stack of punch cards with your instructions, feeding them into the machine, and waiting. During this time, no one else could use the computer, it was one job at a time.

The big goal of Project MAC was to allow multiple programmers to use the same computer simultaneously, executing different instructions at the same time. This concept was called time-sharing.

MIT and Bell Labs cooperated and developed the first operating system to support time-sharing: CTSS (Compatible Time-Sharing System). They wanted to expand this to larger mainframe computers, so they partnered with General Electric (GE), who manufactured these machines. In 1964, they developed the first real OS with time-sharing support called Multics. It also introduced the terminal as a new type of input device.

In the late 1960s, GE and Bell Labs left the project. GE's computer department was bought by Honeywell, which continued the project with MIT and created a commercial version that sold for 25 years.

In 1969, Bell Labs engineers (Dennis Ritchie and Ken Thompson) developed a new OS based on Multics. In 1970, they introduced Unics (later called Unix, the name was a sarcastic play on "Multics," implying it was simpler).

The first two versions of Unix were written in assembly language, which was then translated by an assembler and linker into machine code. The big problem with assembly was that it was tightly coupled to specific processors, meaning you'd need to rewrite Unix for each processor architecture. So Dennis Ritchie decided to create a new programming language: C.

They rebuilt Unix using C. At this time, AT&T owned Bell Labs (now it's Nokia). AT&T declared that Unix was theirs and no one else could touch it, classic monopolization.

AT&T did make one merciful agreement: universities could use Unix for educational purposes. But after AT&T was broken up into smaller companies in 1984, even this stopped. Things got worse.

One person was watching all this and decided to take action: Andrew S. Tanenbaum. In 1987, he created a new Unix-inspired OS called MINIX. It was free for universities and designed to work on Intel chips. It had some issues, occasional crashes and overheating, but this was just the beginning. This was the first time someone made a Unix-like OS outside of AT&T.

The main difference between Unix and MINIX was that MINIX was built on a microkernel architecture. Unix had a larger monolithic kernel, but MINIX separated some modules, for example, device drivers were moved from kernel space to user space.

It's unclear if MINIX was truly open source, but people outside universities wanted access and wanted to contribute and modify it.

Around the same time MINIX was being developed, another person named Richard Stallman started the free software movement based on four freedoms: Freedom to run, Freedom to study, Freedom to modify, and Freedom to share. This led to the GPL license (GNU General Public License), which ensured that if you used something free, your product must also be free. They created the GNU Project, which produced many important tools like the GCC compiler, Bash shell, and more.

But there was one problem: the kernel, the beating heart of the operating system that talks to the hardware, was missing.

Let's leave the USA and cross the Atlantic Ocean. In Finland, a student named Linus Torvalds was stuck at home while his classmates vacationed in Baltim Egypt (kidding). He was frustrated with MINIX, had heard about GPL and GNU, and decided to make something new. "I know what I should do with my life," he thought. As a side hobby project in 1991, he started working on a new kernel (not based on MINIX) and sent an email to his classmates discussing it.

Linus announced Freax (maybe meant "free Unix") with a GPL license. After six months, he released another version and called it Linux. He improved the kernel and integrated many GNU Project tools. He uploaded the source code to the internet (though Git came much later, he initially used FTP). This mini-project became the most widely used OS on Earth.

The penguin mascot (Tux) came from multiple stories: Linus was supposedly bitten by a penguin at a zoo, and he also watched March of the Penguins and was inspired by how they cooperate and share to protect their eggs and each other. Cute and fitting.

...And that's the history intro.

Linux Distributions

Okay... let's install Linux. Which Linux? Wait, really? There are multiple Linuxes?

Here's the deal: the open-source part is the kernel, but different developers take it and add their own packages, libraries, and maybe create a GUI. Others add their own tweaks and features. This leads to many different versions, which we call distributions (or distros for short).

Some examples: Red Hat, Slackware, Debian.

Even distros themselves can be modified with additional features, which creates a version of a version. For example, Debian led to Ubuntu, these are called derivatives.

How many distros and derivatives exist in the world? Many. How many exactly? I said many. Anyone with a computer can create one.

So what's the main difference between these distros, so I know which one is suitable for me? The main differences fall into two categories: philosophical and technical.

One of the biggest technical differences is package management, the system that lets you install software, including the type and format of software itself.

Another difference is configuration files, their locations differ from one distro to another.

We agreed that everything is free, right? Well, you may find some paid versions like Red Hat Enterprise Linux, which charges for features like an additional layer of security, professional support, and guaranteed upgrades. Fedora is also owned by Red Hat and acts as a testing ground (a "backdoor," if you will) for new features before they hit Red Hat Enterprise.

The philosophical part is linked to the functional part. If you're using Linux for research, there are distros with specialized software for that. Maybe you're into ethical hacking, Kali Linux is for you. If you're afraid of switching from another OS, you might like Linux Mint, which even has themes that make it look like Windows.

Okay, which one should I install now? Heh... There are a ton of options and you can install any of them, but my preference is Ubuntu.

Ubuntu is the most popular for development and data engineering. But remember, in all cases, you'll be using the terminal a lot. So install Ubuntu, maybe in dual boot, and keep Windows if possible so you don't regret it later and blame me.

The Terminal

Yes, this is what matters for us. Every distro will come with a default terminal but you can install others if you want. Anyway, open the terminal from the apps or just click Ctrl+Alt+T.

Zoom in using Ctrl+Shift++ or out using Ctrl+-

By default first thing you will see the prompt name@host:path$ which your name @ the machine name then ~ then dollar sign colon then $. After $ you can write your command.

You can change the colors and all preferences and save each for profile.

You can even change the prompt itself as it is just a variable (more on variable later).

Basic Commands

First, everything is case sensitive, so be careful.

[1] echo

This command echoes whatever you write after it.

$ echo "Hello, terminal"

Output:

Hello, terminal

[2] pwd

This prints the current directory.

$ pwd

Output:

/home/mahmoudxyz

[3] cd

This is for changing the directory.

$ cd Desktop

The directory changed with no output, you can check this using pwd.

To go back to the main directory use:

$ cd ~

Or just:

$ cd

Note that this means we are back to /home/mahmoudxyz

To go back to the previous directory (in this case /home) even if you don't know the name, you can use:

$ cd ..

[4] ls

This command outputs the current files and directories (folders).

First let's go to desktop again:

$ cd /home/mahmoudxyz/Desktop

Yes, you can go to a specific dir if you know its path. Note that in Linux we are using / not \ like Windows.

Now let's see what files and directories are in my Desktop:

$ ls

Output:

file1  python  testdir

If you notice that in my case, my terminal supports colors. The blue ones are directories and the grey (maybe black) is the file.

But you may deal with some terminal that doesn't support colors, in this case you can use:

$ ls -F

Output:

file1  python/  testdir/

What ends with / like python/ is a directory otherwise it's a file like file1.

You can see the hidden files using:

$ ls -a

Output:

.  ..  file1  python  testdir  .you-cant-see-me

We saw .you-cant-see-me, but we are not hackers that we saw something hidden, being hidden is more than organizing purpose than actually hiding something.

You can also list the files in the long format using:

$ ls -l

Output:

total 8
-rw-rw-r-- 1 mahmoudxyz mahmoudxyz    0 Nov  2 10:48 file1
drwxrwxr-x 2 mahmoudxyz mahmoudxyz 4096 Oct 16 15:20 python
drwxrwxr-x 2 mahmoudxyz mahmoudxyz 4096 Nov  1 21:45 testdir

Let's take the file1 and analyze the output:

We can also combine these flags/options:

$ ls -l -a -F

Output:

total 16
drwxr-xr-x  4 mahmoudxyz mahmoudxyz 4096 Nov  2 10:53 ./
drwxr-x--- 47 mahmoudxyz mahmoudxyz 4096 Nov  1 21:55 ../
-rw-rw-r--  1 mahmoudxyz mahmoudxyz    0 Nov  2 10:48 file1
drwxrwxr-x  2 mahmoudxyz mahmoudxyz 4096 Oct 16 15:20 python/
drwxrwxr-x  2 mahmoudxyz mahmoudxyz 4096 Nov  1 21:45 testdir/
-rw-rw-r--  1 mahmoudxyz mahmoudxyz    0 Nov  2 10:53 .you-cant-see-me

Or shortly:

$ ls -laF

The same output. The order of options is not important so ls -lFa will work as well.

[5] clear

This cleans your terminal. You can also use shortcut Ctrl+l

[6] mkdir

This makes a new directory.

$ mkdir new-dir

Then let's see the output:

$ ls -F

Output:

file1  new-dir/  python/  testdir/

[7] rmdir

This will remove the directory.

$ rmdir new-dir

Then let's see the output:

$ ls -F

Output:

file1  python/  testdir/

[8] touch

This command is for creating a new file.

$ mkdir new-dir
$ cd new-dir
$ touch file1
$ ls -l

Output:

total 0
-rw-rw-r-- 1 mahmoudxyz mahmoudxyz 0 Nov  2 11:26 file1

You can also make more than one file with:

$ touch file2 file3
$ ls -l

Output:

total 0
-rw-rw-r-- 1 mahmoudxyz mahmoudxyz 0 Nov  2 11:26 file1
-rw-rw-r-- 1 mahmoudxyz mahmoudxyz 0 Nov  2 11:28 file2
-rw-rw-r-- 1 mahmoudxyz mahmoudxyz 0 Nov  2 11:28 file3

In fact touch was created for modifying the timestamp of the file so let's try again:

$ touch file1
$ ls -l

Output:

total 0
-rw-rw-r-- 1 mahmoudxyz mahmoudxyz 0 Nov  2 11:30 file1
-rw-rw-r-- 1 mahmoudxyz mahmoudxyz 0 Nov  2 11:28 file2
-rw-rw-r-- 1 mahmoudxyz mahmoudxyz 0 Nov  2 11:28 file3

What changed? The timestamp of file1. The touch is the easiest way to create a new file, it just changes the timestamp of the file and if it doesn't exist, it will create a new one.

[9] rm

This will remove the file.

$ rm file1
$ ls -l

Output:

total 0
-rw-rw-r-- 1 mahmoudxyz mahmoudxyz 0 Nov  2 11:28 file2
-rw-rw-r-- 1 mahmoudxyz mahmoudxyz 0 Nov  2 11:28 file3

[10] echo & cat (revisited)

Yes again, but this time, it will be used to create a new file with some text inside it.

$ echo "Hello, World" > file1

To output this file we can use:

$ cat file1

Output:

Hello, World

Notes:

• If file1 doesn't exist, it will create a new one.
• If it does exist → it will be overwritten.

To append text instead of overwrite use >>:

$ echo "Hello, Mah" >> file1

To output this file we can use:

$ cat file1

Output:

Hello, World
Hello, Mah

[11] rm -r

Let's go back:

$ cd ..

And then let's try to remove the directory:

$ rmdir new-dir

Output:

rmdir: failed to remove 'new-dir': Directory not empty

In case the directory is not empty, we can use rm that we used for removing a file but this time with a flag -r which means recursively remove everything in the folder.

$ rm -r new-dir

[12] cp

This command is for copying a file.

cp source destination

(you can also rename it while copying it)

For example, let's copy the hosts file:

$ cp /etc/hosts .

The dot . means the current directory. Meaning copy this file from this source to here. You can see the content of the file using cat as before.

[13] man

man is the built-in manual for commands. It contains short descriptions for the command and its options and their functions. It is useful and can be replaced nowadays with online search or even AI.

Try:

$ man ls

And then try:

$ man cd

No manual entry for cd. I don't know why exactly, but it's probably because cd is built into the shell itself and not an external command or maybe programmer choice.

Unix Philosophy

Second System Syndrome: If a software or system succeeds, any similar system that comes after it will likely fail. This is probably a psychological phenomenon, developers constantly compare themselves to the successful system, wanting to be like it but better. The fear of not matching that success often causes failure. Maybe you can succeed if you don't compare yourself to it.

Another thing: when developers started making software for Linux, everything was chaotic and random. This led to the creation of principles to govern development, a philosophy to follow. These principles ensure that when you develop something, you follow the same Unix mentality:

1. Small is Beautiful – Keep programs compact and focused; bloat is the enemy.
2. Each Program Does One Thing Well – Master one task instead of being mediocre at many.
3. Prototype as Soon as Possible – Build it, test it, break it, learn from it, fast iteration wins.
4. Choose Portability Over Efficiency – Code that runs everywhere beats code that's blazing fast on one system.
5. Store Data in Flat Text Files – Text is universal, readable, and easy to parse; proprietary formats lock you in.
6. Use Software Leverage – Don't reinvent the wheel; use existing tools and combine them creatively.
7. Use Shell Scripts to Increase Leverage and Portability – Automate tasks and glue programs together with simple scripts.
8. Avoid Captive User Interfaces – Don't trap users in rigid menus; let them pipe, redirect, and automate.
9. Make Every Program a Filter – Take input, transform it, produce output, programs should be composable building blocks.

These concepts all lead to one fundamental Unix principle: everything is a file. Devices, processes, sockets, treat them all as files for consistency and simplicity.

Not all people follow this now, but the important question is: is it important? I don't know. But still the question is: is it important for you as a data engineer or analyst who will deal with data and different distros and different computers which maybe will be remote? Yes, it is important and very important.

Text Files

It's a bit strange that we are talking about editing text files in 2025. Really, does it matter?

Yes, it matters and it's a big topic in Linux because of what we discussed in the previous section.

There are a lot of editors on Linux like vi, nano and emacs. There is a famous debate between emacs and vim.

You can find vi in almost every distro. The shortcuts for it are many and hard to memorize if you are not dealing with it much, but you can use cheatsheets.

Simply put: vi is just two things, insert mode and command mode. The default when you open a file for the first time is the command mode. To start writing something you have to enter the insert mode by pressing i.

You might wonder why vi uses keyboard letters for navigation instead of arrow keys. Simple answer: arrow keys didn't exist on keyboards when vi was created in 1976. You're the lucky generation with arrow keys, the original vi users had to make do with what they had.

nano on the other hand is more simple and easier to use and edit files with.

Use any editor, probably vi or nano and start practicing on one.

Terminal vs Shell

Terminal ≠ Shell. Let's clear this up.

The shell is the thing that actually interprets your commands. It's the engine doing the work. File manipulation, running programs, printing text. That's all the shell.

The terminal is just the program that opens a window so you can talk to the shell. It's the middleman, the GUI wrapper, the pretty face.

Pipes, Filters and Redirection

Standard Streams

Unix processes use I/O streams to read and write data.

Input stream sources include keyboards, terminals, devices, files, output from other processes, etc.

Unix processes have three standard streams:

• STDIN (0) – Standard Input (data coming in from keyboard, file, etc.)
• STDOUT (1) – Standard Output (normal output going to terminal, file, etc.)
• STDERR (2) – Standard Error (error messages going to terminal, file, etc.)

Example: Try running cat with no arguments, it waits for input from STDIN and echoes it to STDOUT.

• Ctrl+D – Stops the input stream and sends an EOF (End of File) signal to the process.
• Ctrl+C – Sends an INT (Interrupt) signal to the process (i.e., kills the process).

Redirection

Redirection allows you to change the defaults for stdin, stdout, or stderr, sending them to different devices or files using their file descriptors.

File Descriptors

A file descriptor is a reference (or handle) used by the kernel to access a file. Every process gets its own file descriptor table.

Redirect stdin with <

Use the < operator to redirect standard input from a file:

$ wc < textfile

Using Heredocs with <<

Accepts input until a specified delimiter word is reached:

$ cat << EOF
# Type multiple lines here
# Press Enter, then type EOF to end
EOF

Using Herestrings with <<<

Pass a string directly as input:

$ cat <<< "Hello, Linux"

Redirect stdout using > and >>

Overwrite a file with > (or explicitly with 1>):

$ who > file      # Redirect stdout to file (overwrite)
$ cat file        # View the file

Append to a file with >>:

$ whoami >> file  # Append stdout to file
$ cat file        # View the file

Redirect stderr using 2> and 2>>

Redirect error messages to a file:

$ ls /xyz 2> err  # /xyz doesn't exist, error goes to err file
$ cat err         # View the error

Combining stdout and stderr

Redirect both stdout and stderr to the same file:

# Method 1: Redirect stderr to err, then stdout to the same place
$ ls /etc /xyz 2> err 1>&2

# Method 2: Redirect stdout to err, then stderr to the same place
$ ls /etc /xyz 1> err 2>&1

# Method 3: Shorthand for redirecting both
$ ls /etc /xyz &> err

$ cat err  # View both output and errors

Ignoring Error Messages with /dev/null

The black hole of Unix, anything sent here disappears:

$ ls /xyz 2> /dev/null  # Suppress error messages

User and Group Management

It is not complicated. The user here is like any other OS. An account with some permission and can do some operations.

There are three types of users in Linux:

Super user

The administrator that can do anything in the world. It is called root.

• ID from 0 to 999

System user

This represents software and not a real person. Some software may need some access and permissions to do some tasks and operations or maybe install something.

• ID from 0 to 999

Normal user

This is us.

• ID >= 1000

Each user has its ID, shell, environmental vars and home dir.

File Ownership and Permissions

(Content to be added)

More on Navigating the Filesystem

Absolute vs Relative Paths

The root directory (/) is like "C:" in Windows, the top of the filesystem hierarchy.

Absolute path: Starts from root, always begins with /

/home/mahmoudxyz/Documents/notes.txt
/etc/passwd
/usr/bin/python3

Relative path: Starts from your current location

Documents/notes.txt          # Relative to current directory
../Desktop/file.txt          # Go up one level, then into Desktop
../../etc/hosts              # Go up two levels, then into etc

Special directory references:

• . = current directory
• .. = parent directory
• ~ = your home directory
• - = previous directory (used with cd -)

Useful Navigation Commands

ls -lh - List in long format with human-readable sizes

$ ls -lh
-rw-r--r-- 1 mahmoud mahmoud 1.5M Nov 10 14:23 data.csv
-rw-r--r-- 1 mahmoud mahmoud  12K Nov 10 14:25 notes.txt

ls -lhd - Show directory itself, not contents

$ ls -lhd /home/mahmoud
drwxr-xr-x 47 mahmoud mahmoud 4.0K Nov 10 12:00 /home/mahmoud

ls -lR - Recursive listing (all subdirectories)

$ ls -lR
./Documents:
-rw-r--r-- 1 mahmoud mahmoud 1234 Nov 10 14:23 file1.txt

./Documents/Projects:
-rw-r--r-- 1 mahmoud mahmoud 5678 Nov 10 14:25 file2.txt

tree - Visual directory tree (may need to install)

$ tree
.
├── Documents
│   ├── file1.txt
│   └── Projects
│       └── file2.txt
├── Downloads
└── Desktop

stat - Detailed file information

$ stat notes.txt
  File: notes.txt
  Size: 1234       Blocks: 8          IO Block: 4096   regular file
Device: 803h/2051d  Inode: 12345678   Links: 1
Access: 2024-11-10 14:23:45.123456789 +0100
Modify: 2024-11-10 14:23:45.123456789 +0100
Change: 2024-11-10 14:23:45.123456789 +0100

Shows: size, inode number, links, permissions, timestamps

Shell Globbing (Wildcards)

Wildcards let you match multiple files with patterns.

* - Matches any number of any characters (including none)

$ echo *                    # All files in current directory
$ echo *.txt                # All files ending with .txt
$ echo file*                # All files starting with "file"
$ echo *data*               # All files containing "data"

? - Matches exactly one character

$ echo b?at                 # Matches: boat, beat, b1at, b@at
$ echo file?.txt            # Matches: file1.txt, fileA.txt
$ echo ???                  # Matches any 3-character filename

[...] - Matches any character inside brackets

$ echo file[123].txt        # Matches: file1.txt, file2.txt, file3.txt
$ echo [a-z]*               # Files starting with lowercase letter
$ echo [A-Z]*               # Files starting with uppercase letter
$ echo *[0-9]               # Files ending with a digit

[!...] - Matches any character NOT in brackets

$ echo [!a-z]*              # Files NOT starting with lowercase letter
$ echo *[!0-9].txt          # .txt files NOT ending with a digit before extension

Practical examples:

$ ls *.jpg *.png            # All image files (jpg or png)
$ rm temp*                  # Delete all files starting with "temp"
$ cp *.txt backup/          # Copy all text files to backup folder
$ mv file[1-5].txt archive/ # Move file1.txt through file5.txt

File Structure: The Three Components

Every file in Linux consists of three parts:

1. Filename

The human-readable name you see and use.

2. Data Block

The actual content stored on disk, the file's data.

3. Inode (Index Node)

Metadata about the file stored in a data structure. Contains:

• File size
• Owner (UID) and group (GID)
• Permissions
• Timestamps (access, modify, change)
• Number of hard links
• Pointers to data blocks on disk
• NOT the filename (filenames are stored in directory entries)

View inode number:

$ ls -i
12345678 file1.txt
12345679 file2.txt

View detailed inode information:

$ stat file1.txt

Links: Hard Links vs Soft Links

What is a Link?

A link is a way to reference the same file from multiple locations. Think of it like shortcuts in Windows, but with two different types.

Hard Links

Concept: Another filename pointing to the same inode and data.

It's like having two labels on the same box. Both names are equally valid, neither is "original" or "copy."

Create a hard link:

$ ln original.txt hardlink.txt

What happens:

• Both filenames point to the same inode
• Both have equal status (no "original")
• Changing content via either name affects both (same data)
• File size, permissions, content are identical (because they ARE the same file)

Check with ls -i:

$ ls -i
12345678 original.txt
12345678 hardlink.txt    # Same inode number!

What if you delete the original?

$ rm original.txt
$ cat hardlink.txt        # Still works! Data is intact

Why? The data isn't deleted until all hard links are removed. The inode keeps a link count, only when it reaches 0 does the system delete the data.

Limitations of hard links:

• Cannot cross filesystems (different partitions/drives)
• Cannot link to directories (to prevent circular references)
• Both files must be on the same partition

Soft Links (Symbolic Links)

Concept: A special file that points to another filename, like a shortcut in Windows.

The soft link has its own inode, separate from the target file.

Create a soft link:

$ ln -s original.txt softlink.txt

What happens:

• softlink.txt has a different inode
• It contains the path to original.txt
• Reading softlink.txt automatically redirects to original.txt

Check with ls -li:

$ ls -li
12345678 -rw-r--r-- 1 mahmoud mahmoud 100 Nov 10 14:00 original.txt
12345680 lrwxrwxrwx 1 mahmoud mahmoud  12 Nov 10 14:01 softlink.txt -> original.txt

Notice:

• Different inode numbers
• l at the start (link file type)
• -> shows what it points to

What if you delete the original?

$ rm original.txt
$ cat softlink.txt        # Error: No such file or directory

The softlink still exists, but it's now a broken link (points to nothing).

Advantages of soft links:

• Can cross filesystems (different partitions/drives)
• Can link to directories
• Can link to files that don't exist yet (forward reference)

Hard Link vs Soft Link: Summary

When to use each:

Hard links:

• Backup/versioning within same filesystem
• Ensure file persists even if "original" name is deleted
• Save space (no duplicate data)

Soft links:

• Link across different partitions
• Link to directories
• Create shortcuts for convenience
• When you want the link to break if target is moved/deleted (intentional dependency)

Practical Examples

Hard link example:

$ echo "Important data" > data.txt
$ ln data.txt backup.txt              # Create hard link
$ rm data.txt                         # "Original" deleted
$ cat backup.txt                      # Still accessible!
Important data

Soft link example:

$ ln -s /usr/bin/python3 ~/python     # Shortcut to Python
$ ~/python --version                  # Works!
Python 3.10.0
$ rm /usr/bin/python3                 # If Python is removed
$ ~/python --version                  # Link breaks
bash: ~/python: No such file or directory

Link to directory (only soft link):

$ ln -s /var/log/nginx ~/nginx-logs   # Easy access to logs
$ cd ~/nginx-logs                     # Navigate via link
$ pwd                                 # Shows real path
/var/log/nginx

Understanding the Filesystem Hierarchy Standard

Mounting

There's no link between the hierarchy of directories and their location on the disk.

For more details, see: Linux Foundation FHS 3.0

File Management

[1] grep

This command to print lines matching pattern

Let's create a file to try examples on it:

echo -e "root\nhello\nroot\nRoot" >> file

Now let's use grep to search for the word root in this file:

$ grep root file

output:

root
root

You can search for anything excluding the root word:

$ grep -v root file

output:

hello
Root

You can search ingoring the case:

$ grep -i root file

result:

root
root
Root

You can also use REGEX:

$ grep -i r. file

result:

root
root
Root

[2] less

to page through a file (an alternative to more)

-- use with /word to search for a word in the file -- use with ?word to search backwards for a word in the file -- use with n to go to the next occurrence of the word -- use with N to go to the previous occurrence of the word -- use with q to quit the file

[3] diff

compare files line by line

[4] file

determine file type

$ file file
file: ASCII text

[5] find and locate

search for files in a directory hierarchy

[6] head and tail

head - output the first part of files head /usr/share/dict/words - display the first 10 lines of the file /usr/share/dict/words head -n 20 /usr/share/dict/words - display the first 20 lines of the file /usr/share/dict/words

tail - output the last part of files tail /usr/share/dict/words - display the last 10 lines of the file /usr/share/dict/words tail -n 20 /usr/share/dict/words - display the last 20 lines of the file /usr/share/dict/words

[7] mv

mv - move (rename) files mv file1 file2 - rename file1 to file2

mv - move (rename) files mv file1 file2 - rename file1 to file2

[8] cp

cp - copy files and directories cp file1 file2 - copy file1 to file2

[9] tar

archive utility

[10] gzip

[11] mount and unmount

what is the meaning of mounitng

Managing Linux Processes

What is a Process?

When Linux executes a program, it:

1. Reads the file from disk
2. Loads it into memory
3. Reads the instructions inside it
4. Executes them one by one

A process is the running instance of that program. It might be visible in your GUI or running invisibly in the background.

Types of Processes

Processes can be executed from different sources:

By origin:

• Compiled programs (C, C++, Rust, etc.)
• Shell scripts containing commands
• Interpreted languages (Python, Perl, etc.)

By trigger:

• Manually executed by a user
• Scheduled (via cron or systemd timers)
• Triggered by events or other processes

By category:

• System processes - Managed by the kernel
• User processes - Started by users (manually, scheduled, or remotely)

The Process Hierarchy

Every Linux system starts with a parent process that spawns all other processes. This is either:

• init or sysvinit (older systems)
• systemd (modern systems)

The first process gets PID 1 (Process ID 1), even though it's technically branched from the kernel itself (PID 0, which you never see directly).

From PID 1, all other processes branch out in a tree structure. Every process has:

• PID (Process ID) - Its own unique identifier
• PPID (Parent Process ID) - The ID of the process that started it

Viewing Processes

[1] ps - Process Snapshot

Basic usage - current terminal only:

$ ps

Output:

    PID TTY          TIME CMD
  14829 pts/1    00:00:00 bash
  14838 pts/1    00:00:00 ps

This shows only processes running in your current terminal session for your user.

All users' processes:

$ ps -a

Output:

    PID TTY          TIME CMD
   2955 tty2     00:00:00 gnome-session-b
  14971 pts/1    00:00:00 ps

All processes in the system:

$ ps -e

Output:

    PID TTY          TIME CMD
      1 ?        00:00:00 systemd
      2 ?        00:00:00 kthreadd
      3 ?        00:00:00 rcu_gp
    ... (hundreds more)

Note: The ? in the TTY column means the process was started by the kernel and has no controlling terminal.

Detailed process information:

$ ps -l

Output:

F S   UID     PID    PPID  C PRI  NI ADDR SZ WCHAN  TTY          TIME CMD
0 S  1000   14829   14821  0  80   0 -  2865 do_wai pts/1    00:00:00 bash
4 R  1000   15702   14829  0  80   0 -  3445 -      pts/1    00:00:00 ps

Here you can see the PPID (parent process ID). Notice that ps has bash as its parent (the PPID of ps matches the PID of bash).

Most commonly used:

$ ps -efl

This shows all processes with full details - PID, PPID, user, CPU time, memory, and command.

Understanding Daemons

Any system process running in the background typically ends with d (named after "daemon"). Examples:

• systemd - System and service manager
• sshd - SSH server
• httpd or nginx - Web servers
• crond - Job scheduler

Daemons are like Windows services - processes that run in the background, whether they're system or user processes.

[2] pstree - Process Tree Visualization

See the hierarchy of all running processes:

$ pstree

Output:

systemd─┬─ModemManager───3*[{ModemManager}]
        ├─NetworkManager───3*[{NetworkManager}]
        ├─accounts-daemon───3*[{accounts-daemon}]
        ├─avahi-daemon───avahi-daemon
        ├─bluetoothd
        ├─colord───3*[{colord}]
        ├─containerd───15*[{containerd}]
        ├─cron
        ├─cups-browsed───3*[{cups-browsed}]
        ├─cupsd───5*[dbus]
        ├─dbus-daemon
        ├─dockerd───19*[{dockerd}]
        ├─fwupd───5*[{fwupd}]
        ... (continues)

What you're seeing:

• systemd is the parent process (PID 1)
• Everything else branches from it
• Multiple processes run in parallel
• Some processes spawn their own children (like dockerd with 19 threads)

This visualization makes it easy to understand process relationships.

[3] top - Live Process Monitor

Unlike ps (which shows a snapshot), top shows real-time process information:

$ top

You'll see:

• Processes sorted by CPU usage (by default)
• Live updates of CPU and memory consumption
• System load averages
• Running vs sleeping processes

Press q to quit.

Useful top commands while running:

• k - Kill a process (prompts for PID)
• M - Sort by memory usage
• P - Sort by CPU usage
• 1 - Show individual CPU cores
• h - Help

[4] htop - Better Process Monitor

htop is like top but modern, colorful, and more interactive.

Installation (if not already installed):

$ which htop   # Check if installed
$ sudo apt install htop   # Install if needed

Run it:

$ htop

Features:

• Color-coded display
• Mouse support (click to select processes)
• Easy process filtering and searching
• Visual CPU and memory bars
• Tree view of process hierarchy
• Built-in kill/nice/priority management

Navigation:

• Arrow keys to move
• F3 - Search for a process
• F4 - Filter by name
• F5 - Tree view
• F9 - Kill a process
• F10 or q - Quit

Foreground vs Background Processes

Sometimes you only have one terminal and want to run multiple long-running tasks. Background processes let you do this.

Foreground Processes (Default)

When you run a command normally, it runs in the foreground and blocks your terminal:

$ sleep 10

Your terminal is blocked for 10 seconds. You can't type anything until it finishes.

Background Processes

Add & at the end to run in the background:

$ sleep 10 &

Output:

[1] 12345

The terminal is immediately available. The numbers show [job_number] PID.

Managing Jobs

View running jobs:

$ jobs

Output:

[1]+  Running                 sleep 10 &

Bring a background job to foreground:

$ fg

If you have multiple jobs:

$ fg %1   # Bring job 1 to foreground
$ fg %2   # Bring job 2 to foreground

Send current foreground process to background:

1. Press Ctrl+Z (suspends the process)
2. Type bg (resumes it in background)

Example:

$ sleep 25
^Z
[1]+  Stopped                 sleep 25

$ bg
[1]+ sleep 25 &

$ jobs
[1]+  Running                 sleep 25 &

Stopping Processes

Process Signals

The kill command doesn't just "kill" - it sends signals to processes. The process decides how to respond.

Common signals:

Using kill

Syntax:

$ kill -SIGNAL PID

Example - find a process:

$ ps
    PID TTY          TIME CMD
  14829 pts/1    00:00:00 bash
  17584 pts/1    00:00:00 sleep
  18865 pts/1    00:00:00 ps

Try graceful termination first (SIGTERM):

$ kill -SIGTERM 17584

Or use the number:

$ kill -15 17584

Or just use default (SIGTERM is default):

$ kill 17584

If the process ignores SIGTERM, force kill (SIGKILL):

$ kill -SIGKILL 17584

Or:

$ kill -9 17584

Verify it's gone:

$ ps
    PID TTY          TIME CMD
  14829 pts/1    00:00:00 bash
  19085 pts/1    00:00:00 ps
[2]+  Killed                  sleep 10

Why SIGTERM vs SIGKILL?

SIGTERM (15) - Graceful shutdown:

• Process can clean up (save files, close connections)
• Child processes are also terminated properly
• Always try this first

SIGKILL (9) - Immediate death:

• Process cannot ignore or handle this signal
• No cleanup happens
• Can create zombie processes if parent doesn't reap children
• Can cause memory leaks or corrupted files
• Use only as last resort

Zombie Processes

A zombie is a dead process that hasn't been cleaned up by its parent.

What happens:

1. Process finishes execution
2. Kernel marks it as terminated
3. Parent should read the exit status (called "reaping")
4. If parent doesn't reap it, it becomes a zombie

Identifying zombies:

$ ps aux | grep Z

Look for processes with state Z (zombie).

Fixing zombies:

• Kill the parent process (zombies are already dead)
• The parent's death forces the kernel to reclassify zombies under init/systemd, which cleans them up
• Or wait - some zombies disappear when the parent finally checks on them

killall - Kill by Name

Instead of finding PIDs, kill all processes with a specific name:

$ killall sleep

This kills ALL processes named sleep, regardless of their PID.

With signals:

$ killall -SIGTERM firefox
$ killall -9 chrome   # Force kill all Chrome processes

Warning: Be careful with killall - it affects all matching processes, even ones you might not want to kill.

Managing Services with systemctl

Modern Linux systems use systemd to manage services (daemons). The systemctl command controls them.

Service Status

Check if a service is running:

$ systemctl status ssh

Output shows:

• Active/inactive status
• PID of the main process
• Recent log entries
• Memory and CPU usage

Starting and Stopping Services

Start a service:

$ sudo systemctl start nginx

Stop a service:

$ sudo systemctl stop nginx

Restart a service (stop then start):

$ sudo systemctl restart nginx

Reload configuration without restarting:

$ sudo systemctl reload nginx

Enable/Disable Services at Boot

Enable a service to start automatically at boot:

$ sudo systemctl enable ssh

Disable a service from starting at boot:

$ sudo systemctl disable ssh

Enable AND start immediately:

$ sudo systemctl enable --now nginx

Listing Services

List all running services:

$ systemctl list-units --type=service --state=running

List all services (running or not):

$ systemctl list-units --type=service --all

List enabled services:

$ systemctl list-unit-files --type=service --state=enabled

Viewing Logs

See logs for a specific service:

$ journalctl -u nginx

Follow logs in real-time:

$ journalctl -u nginx -f

See only recent logs:

$ journalctl -u nginx --since "10 minutes ago"

Practical Examples

Example 1: Finding and Killing a Hung Process

# Find the process
$ ps aux | grep firefox

# Kill it gracefully
$ kill 12345

# Wait a few seconds, check if still there
$ ps aux | grep firefox

# Force kill if necessary
$ kill -9 12345

Example 2: Running a Long Script in Background

# Start a long-running analysis
$ python analyze_genome.py &

# Check it's running
$ jobs

# Do other work...

# Bring it back to see output
$ fg

Example 3: Checking System Load

# See what's consuming resources
$ htop

# Or check load average
$ uptime

# Or see top CPU processes
$ ps aux --sort=-%cpu | head

Example 4: Restarting a Web Server

# Check status
$ systemctl status nginx

# Restart it
$ sudo systemctl restart nginx

# Check logs if something went wrong
$ journalctl -u nginx -n 50

Summary: Process Management Commands

Shell Scripts (Bash Scripting)

A shell script is simply a collection of commands written in a text file. That's it. Nothing magical.

The original name was "shell script," but when GNU created bash (Bourne Again SHell), the term "bash script" became common.

Why Shell Scripts Matter

1. Automation
If you're typing the same commands repeatedly, write them once in a script.

2. Portability
Scripts work across different Linux machines and distributions (mostly).

3. Scheduling
Combine scripts with cron jobs to run tasks automatically.

4. DRY Principle
Don't Repeat Yourself - write once, run many times.

Important: Nothing new here. Everything you've already learned about Linux commands applies. Shell scripts just let you organize and automate them.

Creating Your First Script

Create a file called first-script.sh:

$ nano first-script.sh

Write some commands:

echo "Hello, World"

Note: The .sh extension doesn't technically matter in Linux (unlike Windows), but it's convention. Use it so humans know it's a shell script.

Making Scripts Executable

Check the current permissions:

$ ls -l first-script.sh

Output:

-rw-rw-r-- 1 mahmoudxyz mahmoudxyz 21 Nov  6 07:21 first-script.sh

Notice: No x (execute) permission. The file isn't executable yet.

Adding Execute Permission

$ chmod +x first-script.sh

Permission options:

• u+x - Execute for user (owner) only
• g+x - Execute for group only
• o+x - Execute for others only
• a+x or just +x - Execute for all (user, group, others)

Check permissions again:

$ ls -l first-script.sh

Output:

-rwxrwxr-x 1 mahmoudxyz mahmoudxyz 21 Nov  6 07:21 first-script.sh

Now we have x for user, group, and others.

Running Shell Scripts

There are two main ways to execute a script:

Method 1: Specify the Shell

$ sh first-script.sh

Or:

$ bash first-script.sh

This explicitly tells which shell to use.

Method 2: Direct Execution

$ ./first-script.sh

Why the ./ ?

Let's try without it:

$ first-script.sh

You'll get an error:

first-script.sh: command not found

Why? When you type a command without a path, the shell searches through directories listed in $PATH looking for that command. Your current directory (.) is usually NOT in $PATH for security reasons.

The ./ explicitly says: "Run the script in the current directory (.), don't search $PATH."

Adding Current Directory to PATH (NOT RECOMMENDED)

You could do this:

$ PATH=.:$PATH

Now first-script.sh would work without ./, but DON'T DO THIS. It's a security risk - you might accidentally execute malicious scripts in your current directory.

Best practices:

1. Use ./script.sh for local scripts
2. Put system-wide scripts in /usr/local/bin (which IS in $PATH)

The Shebang Line

Problem: How does the system know which interpreter to use for your script? Bash? Zsh? Python?

Solution: The shebang (#!) on the first line.

Basic Shebang

#!/bin/bash
echo "Hello, World"

What this means:
"Execute this script using /bin/bash"

When you run ./first-script.sh, the system:

1. Reads the first line
2. Sees #!/bin/bash
3. Runs /bin/bash first-script.sh

Shebang with Other Languages

You can use shebang for any interpreted language:

#!/usr/bin/python3
print("Hello, World")

Now this file runs as a Python script!

The Portable Shebang

Problem: What if bash isn't at /bin/bash? What if python3 is at /usr/local/bin/python3 instead of /usr/bin/python3?

Solution: Use env to find the interpreter:

#!/usr/bin/env bash
echo "Hello, World"

Or for Python:

#!/usr/bin/env python3
print("Hello, World")

How it works:
env searches through $PATH to find the command. The shebang becomes: "Please find (env) where bash is located and execute this script with it."

Why env is better:

• More portable across systems
• Finds interpreters wherever they're installed
• env itself is almost always at /usr/bin/env

Basic Shell Syntax

Command Separators

Semicolon (;) - Run commands sequentially:

$ echo "Hello" ; ls

This runs echo, then runs ls (regardless of whether echo succeeded).

AND (&&) - Run second command only if first succeeds:

$ echo "Hello" && ls

If echo succeeds (exit code 0), then run ls. If it fails, stop.

OR (||) - Run second command only if first fails:

$ false || ls

If false fails (exit code non-zero), then run ls. If it succeeds, stop.

Practical example:

$ cd /some/directory && echo "Changed directory successfully"

Only prints the message if cd succeeded.

$ cd /some/directory || echo "Failed to change directory"

Only prints the message if cd failed.

Variables

Variables store data that you can use throughout your script.

Declaring Variables

#!/bin/bash

# Integer variable
declare -i sum=16

# String variable
declare name="Mahmoud"

# Constant (read-only)
declare -r PI=3.14

# Array
declare -a names=()
names[0]="Alice"
names[1]="Bob"
names[2]="Charlie"

Key points:

• declare -i = integer type
• declare -r = read-only (constant)
• declare -a = array
• You can also just use sum=16 without declare (it works, but less explicit)

Using Variables

Access variables with $:

echo $sum          # Prints: 16
echo $name         # Prints: Mahmoud
echo $PI           # Prints: 3.14

For arrays and complex expressions, use ${}:

echo ${names[0]}   # Prints: Alice
echo ${names[1]}   # Prints: Bob
echo ${names[2]}   # Prints: Charlie

Why ${} matters:

echo "$nameTest"   # Looks for variable called "nameTest" (doesn't exist)
echo "${name}Test" # Prints: MahmoudTest (correct!)

Important Script Options

set -e

What it does: Exit script immediately if any command fails (non-zero exit code).

Why it matters: Prevents cascading errors. If step 1 fails, don't continue to step 2.

Example without set -e:

cd /nonexistent/directory
rm -rf *  # DANGER! This still runs even though cd failed

Example with set -e:

set -e
cd /nonexistent/directory  # Script stops here if this fails
rm -rf *                   # Never executes

Exit Codes

Every command returns an exit code:

• 0 = Success
• Non-zero = Failure (different numbers mean different errors)

Check the last command's exit code:

$ true
$ echo $?   # Prints: 0

$ false
$ echo $?   # Prints: 1

In scripts, explicitly exit with a code:

#!/bin/bash
echo "Script completed successfully"
exit 0  # Return 0 (success) to the calling process

Arithmetic Operations

There are multiple ways to do math in bash. Pick one and stick with it for consistency.

Method 1: $(( )) (Recommended)

#!/bin/bash

num=4
echo $((num * 5))      # Prints: 20
echo $((num + 10))     # Prints: 14
echo $((num ** 2))     # Prints: 16 (exponentiation)

Operators:

• + addition
• - subtraction
• * multiplication
• / integer division
• % modulo (remainder)
• ** exponentiation

Pros: Built into bash, fast, clean syntax
Cons: Integer-only (no decimals)

Method 2: expr

#!/bin/bash

num=4
expr $num + 6      # Prints: 10
expr $num \* 5     # Prints: 20 (note the backslash before *)

Pros: Traditional, works in older shells
Cons: Awkward syntax, needs escaping for *

Method 3: bc (For Floating Point)

#!/bin/bash

echo "4.5 + 2.3" | bc       # Prints: 6.8
echo "10 / 3" | bc -l       # Prints: 3.33333... (-l for decimals)
echo "scale=2; 10/3" | bc   # Prints: 3.33 (2 decimal places)

Pros: Supports floating-point arithmetic
Cons: External program (slower), more complex

My recommendation: Use $(( )) for most cases. Use bc when you need decimals.

Logical Operations and Conditionals

Exit Code Testing

#!/bin/bash

true ; echo $?    # Prints: 0
false ; echo $?   # Prints: 1

Logical Operators

true && echo "True"     # Prints: True (because true succeeds)
false || echo "False"   # Prints: False (because false fails)

Comparison Operators

There are TWO syntaxes for comparisons in bash. Stick to one.

Option 1: [[ ]] with Comparison Operators (Modern, Recommended)

For integers:

[[ 1 -le 2 ]]  # Less than or equal
[[ 3 -ge 2 ]]  # Greater than or equal
[[ 5 -lt 10 ]] # Less than
[[ 8 -gt 4 ]]  # Greater than
[[ 5 -eq 5 ]]  # Equal
[[ 5 -ne 3 ]]  # Not equal

For strings and mixed:

[[ 3 == 3 ]]   # Equal
[[ 3 != 4 ]]   # Not equal
[[ 5 > 3 ]]    # Greater than (lexicographic for strings)
[[ 2 < 9 ]]    # Less than (lexicographic for strings)

Testing the result:

[[ 3 == 3 ]] ; echo $?   # Prints: 0 (true)
[[ 3 != 3 ]] ; echo $?   # Prints: 1 (false)
[[ 5 > 3 ]] ; echo $?    # Prints: 0 (true)

Option 2: test Command (Traditional)

test 1 -le 5 ; echo $?   # Prints: 0 (true)
test 10 -lt 5 ; echo $?  # Prints: 1 (false)

test is equivalent to [ ] (note: single brackets):

[ 1 -le 5 ] ; echo $?    # Same as test

My recommendation: Use [[ ]] (double brackets). It's more powerful and less error-prone than [ ] or test.

File Test Operators

Check file properties:

test -f /etc/hosts ; echo $?     # Does file exist? (0 = yes)
test -d /home ; echo $?           # Is it a directory? (0 = yes)
test -r /etc/shadow ; echo $?    # Do I have read permission? (1 = no)
test -w /tmp ; echo $?            # Do I have write permission? (0 = yes)
test -x /usr/bin/ls ; echo $?    # Is it executable? (0 = yes)

Common file tests:

• -f file exists and is a regular file
• -d directory exists
• -e exists (any type)
• -r readable
• -w writable
• -x executable
• -s file exists and is not empty

Using [[ ]] syntax:

[[ -f /etc/hosts ]] && echo "File exists"
[[ -r /etc/shadow ]] || echo "Cannot read this file"

Positional Parameters (Command-Line Arguments)

When you run a script with arguments, bash provides special variables to access them.

Special Variables

#!/bin/bash

# $0 - Name of the script itself
# $# - Number of command-line arguments
# $* - All arguments as a single string
# $@ - All arguments as separate strings (array-like)
# $1 - First argument
# $2 - Second argument
# $3 - Third argument
# ... and so on

Example Script

#!/bin/bash

echo "Script name: $0"
echo "Total number of arguments: $#"
echo "All arguments: $*"
echo "First argument: $1"
echo "Second argument: $2"

Running it:

$ ./script.sh hello world 123

Output:

Script name: ./script.sh
Total number of arguments: 3
All arguments: hello world 123
First argument: hello
Second argument: world

$* vs $@

$* - Treats all arguments as a single string:

for arg in "$*"; do
    echo $arg
done
# Output: hello world 123 (all as one)

$@ - Treats arguments as separate items:

for arg in "$@"; do
    echo $arg
done
# Output:
# hello
# world
# 123

Recommendation: Use "$@" when looping through arguments.

Functions

Functions let you organize code into reusable blocks.

Basic Function

#!/bin/bash

Hello() {
    echo "Hello Functions!"
}

Hello  # Call the function

Alternative syntax:

function Hello() {
    echo "Hello Functions!"
}

Both work the same. Pick one style and be consistent.

Functions with Return Values

#!/bin/bash

function Hello() {
    echo "Hello Functions!"
    return 0  # Success
}

function GetTimestamp() {
    echo "The time now is $(date +%m/%d/%y' '%R)"
    return 0
}

Hello
echo "Exit code: $?"  # Prints: 0

GetTimestamp

Important: return only returns exit codes (0-255), NOT values like other languages.

To return a value, use echo:

function Add() {
    local result=$(($1 + $2))
    echo $result  # "Return" the value via stdout
}

sum=$(Add 5 3)  # Capture the output
echo "Sum: $sum"  # Prints: Sum: 8

Function Arguments

Functions can take arguments like scripts:

#!/bin/bash

Greet() {
    echo "Hello, $1!"  # $1 is first argument to function
}

Greet "Mahmoud"  # Prints: Hello, Mahmoud!
Greet "World"    # Prints: Hello, World!

Reading User Input

Basic read Command

#!/bin/bash

echo "What is your name?"
read name
echo "Hello, $name!"

How it works:

1. Script displays prompt
2. Waits for user to type and press Enter
3. Stores input in variable name

read with Inline Prompt

#!/bin/bash

read -p "What is your name? " name
echo "Hello, $name!"

-p flag: Display prompt on same line as input

Reading Multiple Variables

#!/bin/bash

read -p "Enter your first and last name: " first last
echo "Hello, $first $last!"

Input: Mahmoud Xyz
Output: Hello, Mahmoud Xyz!

Reading Passwords (Securely)

#!/bin/bash

read -sp "Enter your password: " password
echo ""  # New line after hidden input
echo "Password received (length: ${#password})"

-s flag: Silent mode - doesn't display what user types
-p flag: Inline prompt

Security note: This hides the password from screen, but it's still in memory as plain text. For real password handling, use dedicated tools.

Reading from Files

#!/bin/bash

while read line; do
    echo "Line: $line"
done < /etc/passwd

Reads /etc/passwd line by line.

Best Practices

1. Always use shebang: #!/usr/bin/env bash
2. Use set -e: Stop on errors
3. Use set -u: Stop on undefined variables
4. Use set -o pipefail: Catch errors in pipes
5. Quote variables: Use "$var" not $var (prevents word splitting)
6. Check return codes: Test if commands succeeded
7. Add comments: Explain non-obvious logic
8. Use functions: Break complex scripts into smaller pieces
9. Test thoroughly: Run scripts in safe environment first

The Holy Trinity of Safety

#!/usr/bin/env bash
set -euo pipefail

• -e exit on error
• -u exit on undefined variable
• -o pipefail exit on pipe failures

About Course Materials

These notes contain NO copied course materials. Everything here is my personal understanding and recitation of concepts, synthesized from publicly available resources (bash documentation, shell scripting tutorials, Linux guides).

This is my academic work, how I've processed and reorganized information from legitimate sources. I take full responsibility for any errors in my understanding.

If you believe any content violates copyright, contact me at mahmoudahmedxyz@gmail.com and I'll remove it immediately.

References

[1] Ahmed Sami (Architect @ Microsoft).
Linux for Data Engineers (Arabic – Egyptian Dialect), 11h 30m.
YouTube

Python

This is a quick overview combined with practice problems. Things might appear in a reversed order sometimes - we'll introduce concepts by solving problems and covering tools as needed.

Resources

Free Books:

• Python 101
• Think Python
• A Practical Introduction to Python Programming

If you want to buy:

• Intro to Python for Computer Science and Data Science
• Python Crash Course

Your First Program

print("Hello, World!")

The print() Function

Optional arguments: sep and end

sep (separator) - what goes between values:

print("A", "B", "C")              # A B C (default: space)
print("A", "B", "C", sep="-")     # A-B-C
print(1, 2, 3, sep=" | ")         # 1 | 2 | 3

end - what prints after the line:

print("Hello")
print("World")
# Output:
# Hello
# World

print("Hello", end=" ")
print("World")
# Output: Hello World

Escape Characters

Practice

Variables and Assignment

A variable stores a value in memory so you can use it later.

x = 7
y = 3
total = x + y
print(total)  # 11

Multiple assignment:

x, y, z = 1, 2, 3

Variable Naming Rules

• Must start with letter or underscore
• Can contain letters, numbers, underscores
• Cannot start with number
• Cannot contain spaces
• Cannot use Python keywords (for, if, class, etc.)
• Case sensitive: age, Age, AGE are different

Assignment Operators

Reading Input

name = input("What's your name? ")
print(f"Hello, {name}!")

Converting input:

age = int(input("How old are you? "))
price = float(input("Enter price: $"))

Practice

Basic Data Types

Strings

Text inside quotes:

name = "Mahmoud"
message = 'Hello'

Can use single or double quotes. Strings can contain letters, numbers, spaces, symbols.

Numbers

• int → Whole numbers: 7, 0, -100
• float → Decimals: 3.14, 0.5, -2.7

Boolean

True or false values:

print(5 > 3)        # True
print(2 == 10)      # False
print("a" in "cat") # True

Logical Operators

Practice

Type Checking and Casting

print(type("hello"))  # <class 'str'>
print(type(10))       # <class 'int'>
print(type(3.5))      # <class 'float'>
print(type(True))     # <class 'bool'>

Type casting:

int("10")      # 10
float(5)       # 5.0
str(3.14)      # "3.14"
bool(0)        # False
bool(5)        # True
list("hi")     # ['h', 'i']

Sequences

Strings

Strings are sequences of characters.

Indexing

Indexes start from 0:

name = "Python"
print(name[0])   # P
print(name[3])   # h

String Operations

# Concatenation
"Hello" + " " + "World"  # "Hello World"

# Multiplication
"ha" * 3                 # "hahaha"

# Length
len("Python")            # 6

# Methods
text = "hello"
text.upper()             # "HELLO"
text.replace("h", "j")   # "jello"

Common String Methods

Practice

Lists

Lists can contain different types and are mutable (changeable).

numbers = [1, 2, 3]
mixed = [1, "hello", True]

List Operations

# Accessing
colors = ["red", "blue", "green"]
print(colors[1])  # "blue"

# Modifying (lists ARE mutable!)
colors[1] = "yellow"

# Adding
colors.append("black")          # Add at end
colors.insert(1, "white")       # Add at position

# Removing
del colors[1]                   # Remove by index
value = colors.pop()            # Remove last
colors.remove("red")            # Remove by value

# Sorting
numbers = [3, 1, 2]
numbers.sort()                  # Permanent
sorted(numbers)                 # Temporary

# Other operations
numbers.reverse()               # Reverse in place
len(numbers)                    # Length

Practice

Slicing

Extract portions of sequences: [start:stop:step]

Basic Slicing

numbers = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

numbers[2:5]      # [2, 3, 4]
numbers[:3]       # [0, 1, 2] - from beginning
numbers[5:]       # [5, 6, 7, 8, 9] - to end
numbers[:]        # Copy everything
numbers[::2]      # [0, 2, 4, 6, 8] - every 2nd element

Negative Indices

Count from the end: -1 is last, -2 is second-to-last

numbers[-1]       # 9 - last element
numbers[-3:]      # [7, 8, 9] - last 3 elements
numbers[:-2]      # [0, 1, 2, 3, 4, 5, 6, 7] - all except last 2
numbers[::-1]     # Reverse!

Practice

Control Flow

If Statements

age = 18

if age >= 18:
    print("Adult")
elif age >= 13:
    print("Teen")
else:
    print("Child")

Practice

Loops

For Loops

# Loop through list
for fruit in ["apple", "banana"]:
    print(fruit)

# With index
for i, fruit in enumerate(["apple", "banana"]):
    print(f"{i}: {fruit}")

# Range
for i in range(5):        # 0, 1, 2, 3, 4
    print(i)

for i in range(2, 5):     # 2, 3, 4
    print(i)

for i in range(0, 10, 2): # 0, 2, 4, 6, 8
    print(i)

While Loops

count = 0
while count < 5:
    print(count)
    count += 1

Control Statements

Practice

String & List Exercises

F-Strings (String Formatting)

Modern, clean way to format strings:

name = 'Ahmed'
age = 45
txt = f"My name is {name}, I am {age}"

Number Formatting

pi = 3.14159265359

f'{pi:.2f}'              # '3.14' - 2 decimals
f'{10:03d}'              # '010' - pad with zeros
f'{12345678:,d}'         # '12,345,678' - commas
f'{42:>10d}'             # '        42' - right align
f'{1234.5:>10,.2f}'      # '  1,234.50' - combined

Functions in F-Strings

name = "alice"
f"Hello, {name.upper()}!"        # 'Hello, ALICE!'

numbers = [3, 1, 4]
f"Sum: {sum(numbers)}"           # 'Sum: 8'

String Methods

split() and join()

# Split
text = "one,two,three"
words = text.split(',')          # ['one', 'two', 'three']
text.split()                     # Split on any whitespace

# Join
words = ['one', 'two', 'three']
', '.join(words)                 # 'one, two, three'
''.join(['H','e','l','l','o'])   # 'Hello'

partition()

Splits at first occurrence:

email = "user@example.com"
username, _, domain = email.partition('@')
# username = 'user', domain = 'example.com'

Character Checks

'123'.isdigit()          # True - all digits
'Hello123'.isalnum()     # True - letters and numbers
'hello'.isalpha()        # True - only letters
'hello'.islower()        # True - all lowercase
'HELLO'.isupper()        # True - all uppercase

Two Sum Problem

# Input: nums = [2, 7, 11, 15], target = 9
# Output: [0, 1]  (because 2 + 7 = 9)

Brute Force Solution (O(n²))

nums = [2, 7, 11, 15]
target = 9

for i in range(len(nums)):
    for j in range(i + 1, len(nums)):
        if nums[i] + nums[j] == target:
            print([i, j])

Unpacking with * and **

Unpacking Iterables (*)

# Basic unpacking
numbers = [1, 2, 3]
a, b, c = numbers

# Catch remaining items
first, *middle, last = [1, 2, 3, 4, 5]
# first = 1, middle = [2, 3, 4], last = 5

# In function calls
def add(a, b, c):
    return a + b + c

numbers = [1, 2, 3]
add(*numbers)  # Same as add(1, 2, 3)

# Combining lists
list1 = [1, 2]
list2 = [3, 4]
combined = [*list1, *list2]  # [1, 2, 3, 4]

Unpacking Dictionaries (**)

# Merge dictionaries
defaults = {'color': 'blue', 'size': 'M'}
custom = {'size': 'L'}
final = {**defaults, **custom}
# {'color': 'blue', 'size': 'L'}

# In function calls
def create_user(name, age, city):
    print(f"{name}, {age}, {city}")

data = {'name': 'Bob', 'age': 30, 'city': 'NYC'}
create_user(**data)

Functions

The DRY Principle

Without a function (repetitive):

# Calculating area three times - notice the pattern?
area1 = 10 * 5
print(f"Area 1: {area1}")

area2 = 8 * 6
print(f"Area 2: {area2}")

area3 = 12 * 4
print(f"Area 3: {area3}")

With a function (clean):

def calculate_area(length, width):
    return length * width

print(f"Area 1: {calculate_area(10, 5)}")
print(f"Area 2: {calculate_area(8, 6)}")
print(f"Area 3: {calculate_area(12, 4)}")

Basic Function Syntax

Declaring a Function

def greet():
    print("Hello, World!")

Anatomy:

• def → keyword to start a function
• greet → function name (use descriptive names!)
• () → parentheses for parameters
• : → colon to start the body
• Indented code → what the function does

Calling a Function

def greet():
    print("Hello, World!")

greet()  # Now it runs!
greet()  # You can call it multiple times

Parameters and Arguments

def greet(name):      # 'name' is a parameter
    print(f"Hello, {name}!")

greet("Alice")        # "Alice" is an argument

Multiple parameters:

def add_numbers(a, b):
    result = a + b
    print(f"{a} + {b} = {result}")

add_numbers(5, 3)     # Output: 5 + 3 = 8

Return Values

Functions can give back results using return:

def multiply(a, b):
    return a * b

result = multiply(4, 5)
print(result)  # 20

# Use the result directly in calculations
total = multiply(3, 7) + multiply(2, 4)  # 21 + 8 = 29

Default Arguments

Give parameters default values if no argument is provided:

def power(base, exponent=2):  # exponent defaults to 2
    return base ** exponent

print(power(5))      # 25 (5²)
print(power(5, 3))   # 125 (5³)

Multiple defaults:

def create_profile(name, age=18, country="USA"):
    print(f"{name}, {age} years old, from {country}")

create_profile("Alice")                    # Uses both defaults
create_profile("Bob", 25)                  # Uses country default
create_profile("Charlie", 30, "Canada")    # No defaults used

# ❌ Wrong
def bad(a=5, b):
    pass

# ✅ Correct
def good(b, a=5):
    pass

Variable Number of Arguments

*args (Positional Arguments)

Use when you don't know how many arguments will be passed:

def sum_all(*numbers):
    total = 0
    for num in numbers:
        total += num
    return total

print(sum_all(1, 2, 3))           # 6
print(sum_all(10, 20, 30, 40))    # 100

**kwargs (Keyword Arguments)

Use for named arguments as a dictionary:

def print_info(**details):
    for key, value in details.items():
        print(f"{key}: {value}")

print_info(name="Alice", age=25, city="New York")
# Output:
# name: Alice
# age: 25
# city: New York

Combining Everything

def flexible(required, *args, default="default", **kwargs):
    print(f"Required: {required}")
    print(f"Args: {args}")
    print(f"Default: {default}")
    print(f"Kwargs: {kwargs}")

flexible("Must have", 1, 2, 3, default="Custom", extra="value")

Scope: Local vs Global

Local scope: Variables inside functions only exist inside that function

def calculate():
    result = 10 * 5  # Local variable
    print(result)

calculate()        # 50
print(result)      # ❌ ERROR! result doesn't exist here

Global scope: Variables outside functions can be accessed anywhere

total = 0  # Global variable

def add_to_total(amount):
    global total  # Modify the global variable
    total += amount

add_to_total(10)
print(total)  # 10

Better approach:

def add_to_total(current, amount):
    return current + amount

total = 0
total = add_to_total(total, 10)  # 10
total = add_to_total(total, 5)   # 15

Decomposition

Bad (one giant function):

def process_order(items, customer):
    # Calculate, discount, tax, print - all in one!
    total = sum(item['price'] for item in items)
    if total > 100:
        total *= 0.9
    total *= 1.08
    print(f"Customer: {customer}")
    print(f"Total: ${total:.2f}")

Good (decomposed):

def calculate_subtotal(items):
    return sum(item['price'] for item in items)

def apply_discount(amount):
    return amount * 0.9 if amount > 100 else amount

def add_tax(amount):
    return amount * 1.08

def print_receipt(customer, total):
    print(f"Customer: {customer}")
    print(f"Total: ${total:.2f}")

def process_order(items, customer):
    subtotal = calculate_subtotal(items)
    discounted = apply_discount(subtotal)
    final = add_tax(discounted)
    print_receipt(customer, final)

Benefits: ✅ Easier to understand ✅ Easier to test ✅ Reusable components ✅ Easier to debug

Practice Exercises

rectangle(2, 4)
# Output:
# ****
# ****

• Version A: Modify the original list
• Version B: Return a new list without modifying the original

words = ["hello", "world"]
add_excitement(words)
# words is now ["hello!", "world!"]

sum_digits(123)   # Returns: 6 (1 + 2 + 3)
sum_digits(4567)  # Returns: 22 (4 + 5 + 6 + 7)

first_diff("hello", "world")  # Returns: 0
first_diff("test", "tent")    # Returns: 2
first_diff("same", "same")    # Returns: -1

• Part A: Write a function that randomly places a 2 in an empty spot
• Part B: Write a function that checks if someone has won (returns True/False)

matches("python", "path")  # Returns: 3 (positions 0, 2, 3)

findall("hello", "l")  # Returns: [2, 3]
findall("test", "x")   # Returns: []

change_case("Hello World")  # Returns: "hELLO wORLD"

Challenge Exercises

• Try it with .sort() method
• Try it without using .sort()

merge([1, 3, 5], [2, 4, 6])  # Returns: [1, 2, 3, 4, 5, 6]

verbose(123456)  
# Returns: "one hundred twenty-three thousand, four hundred fifty-six"

base20(0)    # Returns: "A"
base20(20)   # Returns: "BA"
base20(39)   # Returns: "BT"
base20(400)  # Returns: "BAA"

closest([1, 6, 3, 9, 11], 8)  # Returns: 6
closest([5, 10, 15, 20], 12)  # Returns: 10

Higher-Order Functions

Why Do We Need Them?

Imagine you have a list of numbers and you want to:

• Keep only the even numbers
• Keep only numbers greater than 10
• Keep only numbers divisible by 3

You could write three different functions... or write ONE function that accepts different "rules" as parameters!

Worked Example: Filtering Numbers

Step 1: The Problem

We have a list of numbers: [3, 8, 15, 4, 22, 7, 11]

We want to filter them based on different conditions.

Step 2: Without Higher-Order Functions (Repetitive)

# Filter for even numbers
def filter_even(numbers):
    result = []
    for num in numbers:
        if num % 2 == 0:
            result.append(num)
    return result

# Filter for numbers > 10
def filter_large(numbers):
    result = []
    for num in numbers:
        if num > 10:
            result.append(num)
    return result

Step 3: With Higher-Order Function (Smart)

def filter_numbers(numbers, condition):
    """
    Filter numbers based on any condition function.
    
    numbers: list of numbers
    condition: a function that returns True/False
    """
    result = []
    for num in numbers:
        if condition(num):  # Call the function we received!
            result.append(num)
    return result

Step 4: Define Simple Condition Functions

def is_even(n):
    return n % 2 == 0

def is_large(n):
    return n > 10

def is_small(n):
    return n < 10

Step 5: Use It!

numbers = [3, 8, 15, 4, 22, 7, 11]

print(filter_numbers(numbers, is_even))   # [8, 4, 22]
print(filter_numbers(numbers, is_large))  # [15, 22, 11]
print(filter_numbers(numbers, is_small))  # [3, 8, 4, 7]

Practice Exercises

def filter_words(words, condition):
    # Your code here
    pass

def is_long(word):
    return len(word) > 5

def starts_with_a(word):
    return word.lower().startswith('a')

# Test it:
words = ["apple", "cat", "banana", "amazing", "dog"]
print(filter_words(words, is_long))         # Should print: ["banana", "amazing"]
print(filter_words(words, starts_with_a))   # Should print: ["apple", "amazing"]

def transform_numbers(numbers, transformer):
    # Your code here: apply transformer to each number
    pass

def double(n):
    return n * 2

def square(n):
    return n ** 2

# Test it:
nums = [1, 2, 3, 4, 5]
print(transform_numbers(nums, double))   # Should print: [2, 4, 6, 8, 10]
print(transform_numbers(nums, square))   # Should print: [1, 4, 9, 16, 25]

def apply_grading(scores, grade_function):
    # Your code here
    pass

def strict_grade(score):
    if score >= 90:
        return 'A'
    elif score >= 80:
        return 'B'
    else:
        return 'C'

def pass_fail(score):
    return 'Pass' if score >= 60 else 'Fail'

# Test it:
scores = [95, 75, 85, 55]
print(apply_grading(scores, strict_grade))  # Should print: ['A', 'C', 'B', 'C']
print(apply_grading(scores, pass_fail))     # Should print: ['Pass', 'Pass', 'Pass', 'Fail']

Conclusion

Challenge Exercise

Validation functions to create:

• is_valid_dna(seq) - checks if sequence contains only A, C, G, T
• is_long_enough(seq) - checks if sequence is at least 10 characters
• has_start_codon(seq) - checks if sequence starts with "ATG"

sequences = ["ATGCGATCG", "ATGXYZ", "AT", "ATGCCCCCCCCCC"]

# Your solution should work like this:
print(validate_sequences(sequences, is_valid_dna))
# [True, False, True, True]

print(validate_sequences(sequences, is_long_enough))
# [False, False, False, True]

Tuples and Sets

Part 1: Tuples

What is a Tuple?

A tuple is essentially an immutable list. Once created, you cannot change its contents.

# List - mutable (can change)
L = [1, 2, 3]
L[0] = 100  # Works fine

# Tuple - immutable (cannot change)
t = (1, 2, 3)
t[0] = 100  # TypeError: 'tuple' object does not support item assignment

Creating Tuples

# With parentheses
t = (1, 2, 3)

# Without parentheses (comma makes it a tuple)
t = 1, 2, 3

# Single element tuple (comma is required!)
t = (1,)    # This is a tuple
t = (1)     # This is just an integer!

# Empty tuple
t = ()
t = tuple()

# From a list
t = tuple([1, 2, 3])

# From a string
t = tuple("hello")  # ('h', 'e', 'l', 'l', 'o')

Common mistake:

# This is NOT a tuple
x = (5)
print(type(x))  # <class 'int'>

# This IS a tuple
x = (5,)
print(type(x))  # <class 'tuple'>

Accessing Tuple Elements

t = ('a', 'b', 'c', 'd', 'e')

# Indexing (same as lists)
print(t[0])     # 'a'
print(t[-1])    # 'e'

# Slicing
print(t[1:3])   # ('b', 'c')
print(t[:3])    # ('a', 'b', 'c')
print(t[2:])    # ('c', 'd', 'e')

# Length
print(len(t))   # 5

Why Use Tuples?

1. Faster and Less Memory

Tuples are more efficient than lists:

import sys

L = [1, 2, 3, 4, 5]
t = (1, 2, 3, 4, 5)

print(sys.getsizeof(L))  # 104 bytes
print(sys.getsizeof(t))  # 80 bytes (smaller!)

2. Safe - Data Cannot Be Changed

When you want to ensure data stays constant:

# RGB color that shouldn't change
RED = (255, 0, 0)
# RED[0] = 200  # Error! Can't modify

# Coordinates
location = (40.7128, -74.0060)  # New York

3. Can Be Dictionary Keys

Lists cannot be dictionary keys, but tuples can:

# This works
locations = {
    (40.7128, -74.0060): "New York",
    (51.5074, -0.1278): "London"
}
print(locations[(40.7128, -74.0060)])  # New York

# This fails
# locations = {[40.7128, -74.0060]: "New York"}  # TypeError!

4. Return Multiple Values

Functions can return tuples:

def get_stats(numbers):
    return min(numbers), max(numbers), sum(numbers)

low, high, total = get_stats([1, 2, 3, 4, 5])
print(low, high, total)  # 1 5 15

Tuple Unpacking

# Basic unpacking
t = (1, 2, 3)
a, b, c = t
print(a, b, c)  # 1 2 3

# Swap values (elegant!)
x, y = 10, 20
x, y = y, x
print(x, y)  # 20 10

# Unpacking with *
t = (1, 2, 3, 4, 5)
first, *middle, last = t
print(first)   # 1
print(middle)  # [2, 3, 4]
print(last)    # 5

Looping Through Tuples

t = ('a', 'b', 'c')

# Basic loop
for item in t:
    print(item)

# With index
for i, item in enumerate(t):
    print(f"{i}: {item}")

# Loop through list of tuples
points = [(0, 0), (1, 2), (3, 4)]
for x, y in points:
    print(f"x={x}, y={y}")

Tuple Methods

Tuples have only two methods (because they're immutable):

t = (1, 2, 3, 2, 2, 4)

# Count occurrences
print(t.count(2))   # 3

# Find index
print(t.index(3))   # 2

Tuples vs Lists Summary

Tuple Exercises

Exercise 1: Create a tuple with your name, age, and city. Print each element.

Exercise 2: Given t = (1, 2, 3, 4, 5), print the first and last elements.

Exercise 3: Write a function that returns the min, max, and average of a list as a tuple.

Exercise 4: Swap two variables using tuple unpacking.

Exercise 5: Create a tuple from the string "ATGC" and count how many times 'A' appears.

Exercise 6: Given a list of (x, y) coordinates, calculate the distance of each from origin.

Exercise 7: Use a tuple as a dictionary key to store city names by their (latitude, longitude).

Exercise 8: Unpack (1, 2, 3, 4, 5) into first, middle (as list), and last.

Exercise 9: Create a function that returns the quotient and remainder of two numbers as a tuple.

Exercise 10: Loop through [(1, 'a'), (2, 'b'), (3, 'c')] and print each pair.

Exercise 11: Convert a list [1, 2, 3] to a tuple and back to a list.

Exercise 12: Find the index of 'G' in the tuple ('A', 'T', 'G', 'C').

Exercise 13: Create a tuple of tuples representing a 3x3 grid and print the center element.

Exercise 14: Given two tuples, concatenate them into a new tuple.

Exercise 15: Sort a list of (name, score) tuples by score in descending order.

# Exercise 1
person = ("Mahmoud", 25, "Bologna")
print(person[0], person[1], person[2])

# Exercise 2
t = (1, 2, 3, 4, 5)
print(t[0], t[-1])

# Exercise 3
def stats(numbers):
    return min(numbers), max(numbers), sum(numbers)/len(numbers)
print(stats([1, 2, 3, 4, 5]))

# Exercise 4
x, y = 10, 20
x, y = y, x
print(x, y)

# Exercise 5
dna = tuple("ATGC")
print(dna.count('A'))

# Exercise 6
import math
coords = [(3, 4), (0, 5), (1, 1)]
for x, y in coords:
    dist = math.sqrt(x**2 + y**2)
    print(f"({x}, {y}): {dist:.2f}")

# Exercise 7
cities = {
    (40.71, -74.00): "New York",
    (51.51, -0.13): "London"
}
print(cities[(40.71, -74.00)])

# Exercise 8
t = (1, 2, 3, 4, 5)
first, *middle, last = t
print(first, middle, last)

# Exercise 9
def div_mod(a, b):
    return a // b, a % b
print(div_mod(17, 5))  # (3, 2)

# Exercise 10
pairs = [(1, 'a'), (2, 'b'), (3, 'c')]
for num, letter in pairs:
    print(f"{num}: {letter}")

# Exercise 11
L = [1, 2, 3]
t = tuple(L)
L2 = list(t)
print(t, L2)

# Exercise 12
dna = ('A', 'T', 'G', 'C')
print(dna.index('G'))  # 2

# Exercise 13
grid = ((1, 2, 3), (4, 5, 6), (7, 8, 9))
print(grid[1][1])  # 5

# Exercise 14
t1 = (1, 2)
t2 = (3, 4)
t3 = t1 + t2
print(t3)  # (1, 2, 3, 4)

# Exercise 15
scores = [("Alice", 85), ("Bob", 92), ("Charlie", 78)]
sorted_scores = sorted(scores, key=lambda x: x[1], reverse=True)
print(sorted_scores)

Part 2: Sets

What is a Set?

A set is a collection of unique elements with no duplicates. Sets work like mathematical sets.

# Duplicates are automatically removed
S = {1, 2, 2, 3, 3, 3}
print(S)  # {1, 2, 3}

# Unordered - no indexing
# print(S[0])  # TypeError!

Creating Sets

# With curly braces
S = {1, 2, 3, 4, 5}

# From a list (removes duplicates)
S = set([1, 2, 2, 3, 3])
print(S)  # {1, 2, 3}

# From a string
S = set("hello")
print(S)  # {'h', 'e', 'l', 'o'}  (no duplicate 'l')

# Empty set (NOT {} - that's an empty dict!)
S = set()
print(type(S))   # <class 'set'>
print(type({}))  # <class 'dict'>

Adding and Removing Elements

S = {1, 2, 3}

# Add single element
S.add(4)
print(S)  # {1, 2, 3, 4}

# Add multiple elements
S.update([5, 6, 7])
print(S)  # {1, 2, 3, 4, 5, 6, 7}

# Remove element (raises error if not found)
S.remove(7)
print(S)  # {1, 2, 3, 4, 5, 6}

# Discard element (no error if not found)
S.discard(100)  # No error
S.discard(6)
print(S)  # {1, 2, 3, 4, 5}

# Pop random element
x = S.pop()
print(x)  # Some element (unpredictable which one)

# Clear all elements
S.clear()
print(S)  # set()

Membership Testing

Very fast - O(1):

S = {1, 2, 3, 4, 5}

print(3 in S)     # True
print(100 in S)   # False
print(100 not in S)  # True

Looping Through Sets

S = {'a', 'b', 'c'}

# Basic loop
for item in S:
    print(item)

# With enumerate
for i, item in enumerate(S):
    print(f"{i}: {item}")

Note: Sets are unordered - iteration order is not guaranteed!

Set Operations (The Powerful Part!)

Sets support mathematical set operations.

Union: Elements in Either Set

A = {1, 2, 3}
B = {3, 4, 5}

# Using | operator
print(A | B)  # {1, 2, 3, 4, 5}

# Using method
print(A.union(B))  # {1, 2, 3, 4, 5}

Intersection: Elements in Both Sets

A = {1, 2, 3}
B = {3, 4, 5}

# Using & operator
print(A & B)  # {3}

# Using method
print(A.intersection(B))  # {3}

Difference: Elements in A but Not in B

A = {1, 2, 3}
B = {3, 4, 5}

# Using - operator
print(A - B)  # {1, 2}
print(B - A)  # {4, 5}

# Using method
print(A.difference(B))  # {1, 2}

Symmetric Difference: Elements in Either but Not Both

A = {1, 2, 3}
B = {3, 4, 5}

# Using ^ operator
print(A ^ B)  # {1, 2, 4, 5}

# Using method
print(A.symmetric_difference(B))  # {1, 2, 4, 5}

Subset and Superset

A = {1, 2}
B = {1, 2, 3, 4}

# Is A a subset of B?
print(A <= B)        # True
print(A.issubset(B)) # True

# Is B a superset of A?
print(B >= A)          # True
print(B.issuperset(A)) # True

# Proper subset (subset but not equal)
print(A < B)  # True
print(A < A)  # False

Disjoint: No Common Elements

A = {1, 2}
B = {3, 4}
C = {2, 3}

print(A.isdisjoint(B))  # True (no overlap)
print(A.isdisjoint(C))  # False (2 is common)

Set Operations Summary

In-Place Operations

Modify the set directly (note the method names end in _update):

A = {1, 2, 3}
B = {3, 4, 5}

# Union in-place
A |= B  # or A.update(B)
print(A)  # {1, 2, 3, 4, 5}

# Intersection in-place
A = {1, 2, 3}
A &= B  # or A.intersection_update(B)
print(A)  # {3}

# Difference in-place
A = {1, 2, 3}
A -= B  # or A.difference_update(B)
print(A)  # {1, 2}

Practical Examples

Remove Duplicates from List

L = [1, 2, 2, 3, 3, 3, 4]
unique = list(set(L))
print(unique)  # [1, 2, 3, 4]

Find Common Elements

list1 = [1, 2, 3, 4]
list2 = [3, 4, 5, 6]
common = set(list1) & set(list2)
print(common)  # {3, 4}

Find Unique DNA Bases

dna = "ATGCATGCATGC"
bases = set(dna)
print(bases)  # {'A', 'T', 'G', 'C'}

Set Exercises

Exercise 1: Create a set from the list [1, 2, 2, 3, 3, 3] and print it.

Exercise 2: Add the number 10 to a set {1, 2, 3}.

Exercise 3: Remove duplicates from [1, 1, 2, 2, 3, 3, 4, 4].

Exercise 4: Find common elements between {1, 2, 3, 4} and {3, 4, 5, 6}.

Exercise 5: Find elements in {1, 2, 3} but not in {2, 3, 4}.

Exercise 6: Find all unique characters in the string "mississippi".

Exercise 7: Check if {1, 2} is a subset of {1, 2, 3, 4}.

Exercise 8: Find symmetric difference of {1, 2, 3} and {3, 4, 5}.

Exercise 9: Check if two sets {1, 2} and {3, 4} have no common elements.

Exercise 10: Given DNA sequence "ATGCATGC", create set of unique nucleotides.

Exercise 11: Combine sets {1, 2}, {3, 4}, {5, 6} into one set.

Exercise 12: Given two lists of students, find students in both classes.

Exercise 13: Remove element 3 from set {1, 2, 3, 4} safely (no error if missing).

Exercise 14: Create a set of prime numbers less than 20 and check membership of 17.

Exercise 15: Given three sets A, B, C, find elements that are in all three.

# Exercise 1
S = set([1, 2, 2, 3, 3, 3])
print(S)  # {1, 2, 3}

# Exercise 2
S = {1, 2, 3}
S.add(10)
print(S)

# Exercise 3
L = [1, 1, 2, 2, 3, 3, 4, 4]
print(list(set(L)))

# Exercise 4
A = {1, 2, 3, 4}
B = {3, 4, 5, 6}
print(A & B)  # {3, 4}

# Exercise 5
A = {1, 2, 3}
B = {2, 3, 4}
print(A - B)  # {1}

# Exercise 6
print(set("mississippi"))

# Exercise 7
A = {1, 2}
B = {1, 2, 3, 4}
print(A <= B)  # True

# Exercise 8
A = {1, 2, 3}
B = {3, 4, 5}
print(A ^ B)  # {1, 2, 4, 5}

# Exercise 9
A = {1, 2}
B = {3, 4}
print(A.isdisjoint(B))  # True

# Exercise 10
dna = "ATGCATGC"
print(set(dna))  # {'A', 'T', 'G', 'C'}

# Exercise 11
A = {1, 2}
B = {3, 4}
C = {5, 6}
print(A | B | C)  # {1, 2, 3, 4, 5, 6}

# Exercise 12
class1 = ["Alice", "Bob", "Charlie"]
class2 = ["Bob", "Diana", "Charlie"]
print(set(class1) & set(class2))  # {'Bob', 'Charlie'}

# Exercise 13
S = {1, 2, 3, 4}
S.discard(3)  # Safe removal
S.discard(100)  # No error
print(S)

# Exercise 14
primes = {2, 3, 5, 7, 11, 13, 17, 19}
print(17 in primes)  # True

# Exercise 15
A = {1, 2, 3, 4}
B = {2, 3, 4, 5}
C = {3, 4, 5, 6}
print(A & B & C)  # {3, 4}

Summary: When to Use What?

Useful modules

This is planned to be added later

Files and Sys Module

Reading Files

# ✅ Best way - file automatically closes
with open("data.txt", "r") as file:
    content = file.read()
    print(content)

# ❌ Old way - must manually close (don't do this)
file = open("data.txt", "r")
content = file.read()
file.close()  # Easy to forget!

File Modes

# Read
with open("data.txt", "r") as f:
    content = f.read()

# Write (overwrites!)
with open("output.txt", "w") as f:
    f.write("Hello, World!")

# Append (adds to end)
with open("log.txt", "a") as f:
    f.write("New entry\n")

Reading Methods

read() - Entire File

with open("data.txt") as f:
    content = f.read()  # Whole file as string

readline() - One Line at a Time

with open("data.txt") as f:
    first = f.readline()   # First line
    second = f.readline()  # Second line

readlines() - All Lines as List

with open("data.txt") as f:
    lines = f.readlines()  # ['line1\n', 'line2\n', ...]

Looping Through Files

# Best way - memory efficient
with open("data.txt") as f:
    for line in f:
        print(line, end="")  # Line already has \n

# With line numbers
with open("data.txt") as f:
    for i, line in enumerate(f, start=1):
        print(f"{i}: {line}", end="")

# Strip newlines
with open("data.txt") as f:
    for line in f:
        line = line.strip()  # Remove \n
        print(line)

# Process as list
with open("data.txt") as f:
    lines = [line.strip() for line in f]

Writing Files

write() - Single String

with open("output.txt", "w") as f:
    f.write("Hello\n")
    f.write("World\n")

writelines() - List of Strings

lines = ["Line 1\n", "Line 2\n", "Line 3\n"]
with open("output.txt", "w") as f:
    f.writelines(lines)

print() to File

with open("output.txt", "w") as f:
    print("Hello, World!", file=f)
    print("Another line", file=f)

Processing Lines

Splitting

# By delimiter
line = "name,age,city"
parts = line.split(",")  # ['name', 'age', 'city']

# By whitespace (default)
line = "John   25   NYC"
parts = line.split()  # ['John', '25', 'NYC']

# With max splits
line = "a,b,c,d,e"
parts = line.split(",", 2)  # ['a', 'b', 'c,d,e']

Joining

words = ['Hello', 'World']
sentence = " ".join(words)  # "Hello World"

lines = ['line1', 'line2', 'line3']
content = "\n".join(lines)

Processing CSV Data

with open("data.csv") as f:
    for line in f:
        parts = line.strip().split(",")
        name, age, city = parts
        print(f"{name} is {age} from {city}")

The sys Module

Command Line Arguments

import sys

print(sys.argv)  # List of all arguments
# python script.py hello world
# Output: ['script.py', 'hello', 'world']

print(sys.argv[0])  # Script name
print(sys.argv[1])  # First argument
print(len(sys.argv))  # Number of arguments

Basic Argument Handling

import sys

if len(sys.argv) < 2:
    print("Usage: python script.py <filename>")
    sys.exit(1)

filename = sys.argv[1]
print(f"Processing: {filename}")

Processing Multiple Arguments

import sys

# python script.py file1.txt file2.txt file3.txt
for filename in sys.argv[1:]:  # Skip script name
    print(f"Processing: {filename}")

Argument Validation

import sys
import os

def main():
    # Check argument count
    if len(sys.argv) != 3:
        print("Usage: python script.py <input> <output>")
        sys.exit(1)
    
    input_file = sys.argv[1]
    output_file = sys.argv[2]
    
    # Check if input exists
    if not os.path.exists(input_file):
        print(f"Error: {input_file} not found")
        sys.exit(1)
    
    # Check if output exists
    if os.path.exists(output_file):
        response = input(f"{output_file} exists. Overwrite? (y/n): ")
        if response.lower() != 'y':
            print("Aborted")
            sys.exit(0)
    
    # Process files
    process(input_file, output_file)

if __name__ == "__main__":
    main()

Standard Streams

stdin, stdout, stderr

import sys

# Read from stdin
line = sys.stdin.readline()

# Write to stdout (like print)
sys.stdout.write("Hello\n")

# Write to stderr (for errors)
sys.stderr.write("Error: failed\n")

Reading from Pipe

# In terminal
cat data.txt | python script.py
echo "Hello" | python script.py

# script.py
import sys

for line in sys.stdin:
    print(f"Received: {line.strip()}")

Exit Codes

import sys

# Exit with success
sys.exit(0)

# Exit with error
sys.exit(1)

# Exit with message
sys.exit("Error: something went wrong")

Useful sys Attributes

import sys

# Python version
print(sys.version)         # '3.10.0 (default, ...)'
print(sys.version_info)    # sys.version_info(major=3, ...)

# Platform
print(sys.platform)        # 'linux', 'darwin', 'win32'

# Module search paths
print(sys.path)

# Maximum integer
print(sys.maxsize)

# Default encoding
print(sys.getdefaultencoding())  # 'utf-8'

Building Command Line Tools

Simple Script Template

#!/usr/bin/env python3
"""Simple command line tool."""

import sys
import os

def print_usage():
    print("Usage: python tool.py <input_file>")
    print("Options:")
    print("  -h, --help    Show help")
    print("  -v, --verbose Verbose output")

def main():
    # Parse arguments
    if len(sys.argv) < 2 or sys.argv[1] in ['-h', '--help']:
        print_usage()
        sys.exit(0)
    
    verbose = '-v' in sys.argv or '--verbose' in sys.argv
    
    # Get input file
    input_file = None
    for arg in sys.argv[1:]:
        if not arg.startswith('-'):
            input_file = arg
            break
    
    if not input_file:
        print("Error: No input file", file=sys.stderr)
        sys.exit(1)
    
    if not os.path.exists(input_file):
        print(f"Error: {input_file} not found", file=sys.stderr)
        sys.exit(1)
    
    # Process
    if verbose:
        print(f"Processing {input_file}...")
    
    with open(input_file) as f:
        for line in f:
            print(line.strip())
    
    if verbose:
        print("Done!")

if __name__ == "__main__":
    main()

Word Count Tool

#!/usr/bin/env python3
import sys

def count_file(filename):
    lines = words = chars = 0
    with open(filename) as f:
        for line in f:
            lines += 1
            words += len(line.split())
            chars += len(line)
    return lines, words, chars

def main():
    if len(sys.argv) < 2:
        print("Usage: python wc.py <file1> [file2] ...")
        sys.exit(1)
    
    total_l = total_w = total_c = 0
    
    for filename in sys.argv[1:]:
        try:
            l, w, c = count_file(filename)
            print(f"{l:8} {w:8} {c:8} {filename}")
            total_l += l
            total_w += w
            total_c += c
        except FileNotFoundError:
            print(f"Error: {filename} not found", file=sys.stderr)
    
    if len(sys.argv) > 2:
        print(f"{total_l:8} {total_w:8} {total_c:8} total")

if __name__ == "__main__":
    main()

FASTA Sequence Counter

#!/usr/bin/env python3
import sys

def process_fasta(filename):
    sequences = 0
    total_bases = 0
    
    with open(filename) as f:
        for line in f:
            line = line.strip()
            if line.startswith(">"):
                sequences += 1
            else:
                total_bases += len(line)
    
    return sequences, total_bases

def main():
    if len(sys.argv) != 2:
        print("Usage: python fasta_count.py <file.fasta>")
        sys.exit(1)
    
    filename = sys.argv[1]
    
    try:
        seqs, bases = process_fasta(filename)
        print(f"Sequences: {seqs}")
        print(f"Total bases: {bases}")
        print(f"Average: {bases/seqs:.1f}")
    except FileNotFoundError:
        print(f"Error: {filename} not found", file=sys.stderr)
        sys.exit(1)

if __name__ == "__main__":
    main()

File Path Operations

import os

# Join paths (cross-platform)
path = os.path.join("folder", "subfolder", "file.txt")

# Get filename
os.path.basename("/path/to/file.txt")  # "file.txt"

# Get directory
os.path.dirname("/path/to/file.txt")   # "/path/to"

# Split extension
name, ext = os.path.splitext("data.txt")  # "data", ".txt"

# Check existence
os.path.exists("file.txt")    # True/False
os.path.isfile("file.txt")    # True if file
os.path.isdir("folder")       # True if directory

# Get file size
os.path.getsize("file.txt")   # Size in bytes

# Get absolute path
os.path.abspath("file.txt")

Practice Exercises

Quick Reference

Best Practices

Solution Hints

Recursive Functions

Classic example: Factorial (5! = 5 × 4 × 3 × 2 × 1)

def factorial(n):
    # Base case: stop condition
    if n == 0 or n == 1:
        return 1
    
    # Recursive case: call itself   
    return n * factorial(n - 1)

print(factorial(5))  # 120

How it works:

factorial(5) = 5 × factorial(4)
             = 5 × (4 × factorial(3))
             = 5 × (4 × (3 × factorial(2)))
             = 5 × (4 × (3 × (2 × factorial(1))))
             = 5 × (4 × (3 × (2 × 1)))
             = 120

Key parts of recursion:

Another example: Countdown

def countdown(n):
    if n == 0:
        print("Blast off!")
        return
    print(n)
    countdown(n - 1)

countdown(3)
# Output: 3, 2, 1, Blast off!

Python Exceptions

Errors vs Bugs vs Exceptions

Syntax Errors

Errors in your code before it runs. Python can't even understand what you wrote.

# Missing colon
if True
    print("Hello")  # SyntaxError: expected ':'

# Unclosed parenthesis
print("Hello"  # SyntaxError: '(' was never closed

Fix: Correct the syntax. Python tells you exactly where the problem is.

Bugs

Your code runs, but it does the wrong thing. No error message - just incorrect behavior.

# Bug: wrong formula
def circle_area(radius):
    return 2 * 3.14 * radius  # Wrong! This is circumference, not area

print(circle_area(5))  # Returns 31.4, should be 78.5

Why "bug"? Legend says early computers had actual insects causing problems. The term stuck.

Fix: Debug your code - find and fix the logic error.

Exceptions

Errors that occur during execution. The code is syntactically correct, but something goes wrong at runtime.

# Runs fine until...
x = 10 / 0  # ZeroDivisionError: division by zero

# Or...
my_list = [1, 2, 3]
print(my_list[10])  # IndexError: list index out of range

Fix: Handle the exception or prevent the error condition.

What is an Exception?

An exception is Python's way of saying "something unexpected happened and I can't continue."

When an exception occurs:

1. Python stops normal execution
2. Creates an exception object with error details
3. Looks for code to handle it
4. If no handler found, program crashes with traceback

# Exception in action
print("Start")
x = 10 / 0  # Exception here!
print("End")  # Never reached

# Output:
# Start
# Traceback (most recent call last):
#   File "example.py", line 2, in <module>
#     x = 10 / 0
# ZeroDivisionError: division by zero

Common Exceptions

# ZeroDivisionError
10 / 0

# TypeError - wrong type
"hello" + 5

# ValueError - right type, wrong value
int("hello")

# IndexError - list index out of range
[1, 2, 3][10]

# KeyError - dictionary key not found
{'a': 1}['b']

# FileNotFoundError
open("nonexistent.txt")

# AttributeError - object has no attribute
"hello".append("!")

# NameError - variable not defined
print(undefined_variable)

# ImportError - module not found
import nonexistent_module

Handling Exceptions

Basic try/except

try:
    x = 10 / 0
except:
    print("Something went wrong!")

# Output: Something went wrong!

Problem: This catches ALL exceptions - even ones you didn't expect. Not recommended.

Catching Specific Exceptions (Recommended)

try:
    x = 10 / 0
except ZeroDivisionError:
    print("Cannot divide by zero!")

# Output: Cannot divide by zero!

Catching Multiple Specific Exceptions

try:
    value = int(input("Enter a number: "))
    result = 10 / value
except ValueError:
    print("That's not a valid number!")
except ZeroDivisionError:
    print("Cannot divide by zero!")

Catching Multiple Exceptions Together

try:
    # Some risky code
    pass
except (ValueError, TypeError):
    print("Value or Type error occurred!")

Getting Exception Details

try:
    x = 10 / 0
except ZeroDivisionError as e:
    print(f"Error: {e}")
    print(f"Type: {type(e).__name__}")

# Output:
# Error: division by zero
# Type: ZeroDivisionError

The Complete try/except/else/finally

try:
    # Code that might raise an exception
    result = 10 / 2
except ZeroDivisionError:
    # Runs if exception occurs
    print("Cannot divide by zero!")
else:
    # Runs if NO exception occurs
    print(f"Result: {result}")
finally:
    # ALWAYS runs, exception or not
    print("Cleanup complete")

# Output:
# Result: 5.0
# Cleanup complete

When to Use Each Part

finally is Guaranteed

def risky_function():
    try:
        return 10 / 0
    except ZeroDivisionError:
        return "Error!"
    finally:
        print("This ALWAYS prints!")

result = risky_function()
# Output: This ALWAYS prints!
# result = "Error!"

Best Practices

1. Be Specific - Don't Catch Everything

# BAD - catches everything, hides bugs
try:
    do_something()
except:
    pass

# GOOD - catches only what you expect
try:
    do_something()
except ValueError:
    handle_value_error()

2. Don't Silence Exceptions Without Reason

# BAD - silently ignores errors
try:
    important_operation()
except Exception:
    pass  # What went wrong? We'll never know!

# GOOD - at least log it
try:
    important_operation()
except Exception as e:
    print(f"Error occurred: {e}")
    # or use logging.error(e)

3. Use else for Code That Depends on try Success

# Less clear
try:
    file = open("data.txt")
    content = file.read()
    process(content)
except FileNotFoundError:
    print("File not found")

# More clear - separate "risky" from "safe" code
try:
    file = open("data.txt")
except FileNotFoundError:
    print("File not found")
else:
    content = file.read()
    process(content)

4. Use finally for Cleanup

file = None
try:
    file = open("data.txt")
    content = file.read()
except FileNotFoundError:
    print("File not found")
finally:
    if file:
        file.close()  # Always close, even if error

# Even better - use context manager
with open("data.txt") as file:
    content = file.read()  # Automatically closes!

5. Catch Exceptions at the Right Level

# Don't catch too early
def read_config():
    # Let the caller handle missing file
    with open("config.txt") as f:
        return f.read()

# Catch at appropriate level
def main():
    try:
        config = read_config()
    except FileNotFoundError:
        print("Config file missing, using defaults")
        config = get_defaults()

Raising Exceptions

Use raise to throw your own exceptions:

def divide(a, b):
    if b == 0:
        raise ValueError("Cannot divide by zero!")
    return a / b

try:
    result = divide(10, 0)
except ValueError as e:
    print(e)  # Cannot divide by zero!

Re-raising Exceptions

try:
    risky_operation()
except ValueError:
    print("Logging this error...")
    raise  # Re-raise the same exception

Built-in Exception Hierarchy

All exceptions inherit from BaseException. Here's the hierarchy:

BaseException
├── SystemExit
├── KeyboardInterrupt
├── GeneratorExit
└── Exception
    ├── StopIteration
    ├── ArithmeticError
    │   ├── FloatingPointError
    │   ├── OverflowError
    │   └── ZeroDivisionError
    ├── AssertionError
    ├── AttributeError
    ├── BufferError
    ├── EOFError
    ├── ImportError
    │   └── ModuleNotFoundError
    ├── LookupError
    │   ├── IndexError
    │   └── KeyError
    ├── MemoryError
    ├── NameError
    │   └── UnboundLocalError
    ├── OSError
    │   ├── FileExistsError
    │   ├── FileNotFoundError
    │   ├── IsADirectoryError
    │   ├── NotADirectoryError
    │   ├── PermissionError
    │   └── TimeoutError
    ├── ReferenceError
    ├── RuntimeError
    │   ├── NotImplementedError
    │   └── RecursionError
    ├── SyntaxError
    │   └── IndentationError
    │       └── TabError
    ├── TypeError
    └── ValueError
        └── UnicodeError
            ├── UnicodeDecodeError
            ├── UnicodeEncodeError
            └── UnicodeTranslateError

Why Hierarchy Matters

Catching a parent catches all children:

# Catches ZeroDivisionError, OverflowError, FloatingPointError
try:
    result = 10 / 0
except ArithmeticError:
    print("Math error!")

# Catches IndexError and KeyError
try:
    my_list[100]
except LookupError:
    print("Lookup failed!")

Tip: Catch Exception instead of bare except: - it doesn't catch KeyboardInterrupt or SystemExit.

# Better than bare except
try:
    do_something()
except Exception as e:
    print(f"Error: {e}")

User-Defined Exceptions

Create custom exceptions by inheriting from Exception:

Basic Custom Exception

class InvalidDNAError(Exception):
    """Raised when DNA sequence contains invalid characters"""
    pass

def validate_dna(sequence):
    valid_bases = set("ATGC")
    for base in sequence.upper():
        if base not in valid_bases:
            raise InvalidDNAError(f"Invalid base: {base}")
    return True

try:
    validate_dna("ATGXCCC")
except InvalidDNAError as e:
    print(f"Invalid DNA: {e}")

Custom Exception with Attributes

class InsufficientFundsError(Exception):
    """Raised when account has insufficient funds"""
    
    def __init__(self, balance, amount):
        self.balance = balance
        self.amount = amount
        self.shortage = amount - balance
        super().__init__(
            f"Cannot withdraw ${amount}. "
            f"Balance: ${balance}. "
            f"Short by: ${self.shortage}"
        )

class BankAccount:
    def __init__(self, balance):
        self.balance = balance
    
    def withdraw(self, amount):
        if amount > self.balance:
            raise InsufficientFundsError(self.balance, amount)
        self.balance -= amount
        return amount

# Usage
account = BankAccount(100)
try:
    account.withdraw(150)
except InsufficientFundsError as e:
    print(e)
    print(f"You need ${e.shortage} more")

# Output:
# Cannot withdraw $150. Balance: $100. Short by: $50
# You need $50 more

Exception Hierarchy for Your Project

# Base exception for your application
class BioinformaticsError(Exception):
    """Base exception for bioinformatics operations"""
    pass

# Specific exceptions
class SequenceError(BioinformaticsError):
    """Base for sequence-related errors"""
    pass

class InvalidDNAError(SequenceError):
    """Invalid DNA sequence"""
    pass

class InvalidRNAError(SequenceError):
    """Invalid RNA sequence"""
    pass

class AlignmentError(BioinformaticsError):
    """Sequence alignment failed"""
    pass

# Now you can catch at different levels
try:
    process_sequence()
except InvalidDNAError:
    print("DNA issue")
except SequenceError:
    print("Some sequence issue")
except BioinformaticsError:
    print("General bioinformatics error")

Exercises

Exercise 1: Write code that catches a ZeroDivisionError and prints a friendly message.

Exercise 2: Ask user for a number, handle both ValueError (not a number) and ZeroDivisionError (if dividing by it).

Exercise 3: Write a function that opens a file and handles FileNotFoundError.

Exercise 4: Create a function that takes a list and index, returns the element, handles IndexError.

Exercise 5: Write code that handles KeyError when accessing a dictionary.

Exercise 6: Create a custom NegativeNumberError and raise it if a number is negative.

Exercise 7: Write a function that converts string to int, handling ValueError, and returns 0 on failure.

Exercise 8: Use try/except/else/finally to read a file and ensure it's always closed.

Exercise 9: Create a custom InvalidAgeError with min and max age attributes.

Exercise 10: Write a function that validates an email (must contain @), raise ValueError if invalid.

Exercise 11: Handle multiple exceptions: TypeError, ValueError, ZeroDivisionError in one block.

Exercise 12: Create a hierarchy: ValidationError → EmailError, PhoneError.

Exercise 13: Re-raise an exception after logging it.

Exercise 14: Create a InvalidSequenceError for DNA validation with the invalid character as attribute.

Exercise 15: Write a "safe divide" function that returns None on any error instead of crashing.

# Exercise 1
try:
    result = 10 / 0
except ZeroDivisionError:
    print("Cannot divide by zero!")

# Exercise 2
try:
    num = int(input("Enter a number: "))
    result = 100 / num
    print(f"100 / {num} = {result}")
except ValueError:
    print("That's not a valid number!")
except ZeroDivisionError:
    print("Cannot divide by zero!")

# Exercise 3
def read_file(filename):
    try:
        with open(filename) as f:
            return f.read()
    except FileNotFoundError:
        print(f"File '{filename}' not found")
        return None

# Exercise 4
def safe_get(lst, index):
    try:
        return lst[index]
    except IndexError:
        print(f"Index {index} out of range")
        return None

# Exercise 5
d = {'a': 1, 'b': 2}
try:
    value = d['c']
except KeyError:
    print("Key not found!")
    value = None

# Exercise 6
class NegativeNumberError(Exception):
    pass

def check_positive(n):
    if n < 0:
        raise NegativeNumberError(f"{n} is negative!")
    return n

# Exercise 7
def safe_int(s):
    try:
        return int(s)
    except ValueError:
        return 0

# Exercise 8
file = None
try:
    file = open("data.txt")
    content = file.read()
except FileNotFoundError:
    print("File not found")
    content = ""
else:
    print("File read successfully")
finally:
    if file:
        file.close()
    print("Cleanup done")

# Exercise 9
class InvalidAgeError(Exception):
    def __init__(self, age, min_age=0, max_age=150):
        self.age = age
        self.min_age = min_age
        self.max_age = max_age
        super().__init__(f"Age {age} not in range [{min_age}, {max_age}]")

# Exercise 10
def validate_email(email):
    if '@' not in email:
        raise ValueError(f"Invalid email: {email} (missing @)")
    return True

# Exercise 11
try:
    # risky code
    pass
except (TypeError, ValueError, ZeroDivisionError) as e:
    print(f"Error: {e}")

# Exercise 12
class ValidationError(Exception):
    pass

class EmailError(ValidationError):
    pass

class PhoneError(ValidationError):
    pass

# Exercise 13
try:
    result = 10 / 0
except ZeroDivisionError:
    print("Logging: Division by zero occurred")
    raise

# Exercise 14
class InvalidSequenceError(Exception):
    def __init__(self, sequence, invalid_char):
        self.sequence = sequence
        self.invalid_char = invalid_char
        super().__init__(f"Invalid character '{invalid_char}' in sequence")

def validate_dna(seq):
    for char in seq:
        if char not in "ATGC":
            raise InvalidSequenceError(seq, char)
    return True

# Exercise 15
def safe_divide(a, b):
    try:
        return a / b
    except Exception:
        return None

print(safe_divide(10, 2))   # 5.0
print(safe_divide(10, 0))   # None
print(safe_divide("a", 2))  # None

Summary

Best Practices:

1. Catch specific exceptions, not bare except:
2. Don't silence exceptions without reason
3. Use finally for cleanup
4. Create custom exceptions for your domain
5. Build exception hierarchies for complex projects

Debugging

Theory

PyCharm Debug Tutorial

Using the IDLE Debugger

Python Dictionaries

What is a Dictionary?

A dictionary stores data as key-value pairs.

# Basic structure
student = {'name': 'Alex', 'age': 20, 'major': 'CS'}

# Access by key
print(student['name'])   # Alex
print(student['age'])    # 20

Creating Dictionaries

# Empty dictionary
empty = {}

# With initial values
person = {'name': 'Alex', 'age': 20}

# Using dict() constructor
person = dict(name='Alex', age=20)

Basic Operations

Adding and Modifying

student = {'name': 'Alex', 'age': 20}

# Add new key
student['major'] = 'CS'
print(student)  # {'name': 'Alex', 'age': 20, 'major': 'CS'}

# Modify existing value
student['age'] = 21
print(student)  # {'name': 'Alex', 'age': 21, 'major': 'CS'}

Deleting

student = {'name': 'Alex', 'age': 20, 'major': 'CS'}

# Delete specific key
del student['major']
print(student)  # {'name': 'Alex', 'age': 20}

# Remove and return value
age = student.pop('age')
print(age)      # 20
print(student)  # {'name': 'Alex'}

Getting Values Safely

student = {'name': 'Alex', 'age': 20}

# Direct access - raises error if key missing
print(student['name'])      # Alex
# print(student['grade'])   # KeyError!

# Safe access with .get() - returns None if missing
print(student.get('name'))   # Alex
print(student.get('grade'))  # None

# Provide default value
print(student.get('grade', 'N/A'))  # N/A

Useful Methods

student = {'name': 'Alex', 'age': 20, 'major': 'CS'}

# Get all keys
print(student.keys())    # dict_keys(['name', 'age', 'major'])

# Get all values
print(student.values())  # dict_values(['Alex', 20, 'CS'])

# Get all key-value pairs
print(student.items())   # dict_items([('name', 'Alex'), ('age', 20), ('major', 'CS')])

# Get length
print(len(student))      # 3

Membership Testing

Use in to check if a key exists (not value!):

student = {'name': 'Alex', 'age': 20}

# Check if key exists
print('name' in student)     # True
print('grade' in student)    # False

# Check if key does NOT exist
print('grade' not in student)  # True

# To check values, use .values()
print('Alex' in student.values())  # True
print(20 in student.values())      # True

Important: Checking in on a dictionary is O(1) - instant! This is why dictionaries are so powerful.

Looping Through Dictionaries

Loop Over Keys (Default)

student = {'name': 'Alex', 'age': 20, 'major': 'CS'}

# Default: loops over keys
for key in student:
    print(key)
# name
# age
# major

# Explicit (same result)
for key in student.keys():
    print(key)

Loop Over Values

student = {'name': 'Alex', 'age': 20, 'major': 'CS'}

for value in student.values():
    print(value)
# Alex
# 20
# CS

Loop Over Keys and Values Together

student = {'name': 'Alex', 'age': 20, 'major': 'CS'}

for key, value in student.items():
    print(f"{key}: {value}")
# name: Alex
# age: 20
# major: CS

Loop With Index Using enumerate()

student = {'name': 'Alex', 'age': 20, 'major': 'CS'}

for index, key in enumerate(student):
    print(f"{index}: {key} = {student[key]}")
# 0: name = Alex
# 1: age = 20
# 2: major = CS

# Or with items()
for index, (key, value) in enumerate(student.items()):
    print(f"{index}: {key} = {value}")

Dictionary Order

Python 3.7+: Dictionaries maintain insertion order.

# Items stay in the order you add them
d = {}
d['first'] = 1
d['second'] = 2
d['third'] = 3

for key in d:
    print(key)
# first
# second
# third  (guaranteed order!)

Note: Before Python 3.7, dictionary order was not guaranteed. If you need to support older Python, don't rely on order.

Important: While keys maintain insertion order, this doesn't mean dictionaries are sorted. They just remember the order you added things.

# Not sorted - just insertion order
d = {'c': 3, 'a': 1, 'b': 2}
print(list(d.keys()))  # ['c', 'a', 'b'] - insertion order, not alphabetical

Complex Values

Lists as Values

student = {
    'name': 'Alex',
    'courses': ['Math', 'Physics', 'CS']
}

# Access list items
print(student['courses'][0])  # Math

# Modify list
student['courses'].append('Biology')
print(student['courses'])  # ['Math', 'Physics', 'CS', 'Biology']

Nested Dictionaries

students = {
    1: {'name': 'Alex', 'age': 20},
    2: {'name': 'Maria', 'age': 22},
    3: {'name': 'Jordan', 'age': 21}
}

# Access nested values
print(students[1]['name'])  # Alex
print(students[2]['age'])   # 22

# Modify nested values
students[3]['age'] = 22

# Add new entry
students[4] = {'name': 'Casey', 'age': 19}

Why Dictionaries Are Fast: Hashing

Dictionaries use hashing to achieve O(1) lookup time.

How it works:

1. When you add a key, Python computes a hash (a number) from the key
2. This hash tells Python exactly where to store the value in memory
3. When you look up the key, Python computes the same hash and goes directly to that location

Result: Looking up a key takes the same time whether your dictionary has 10 items or 10 million items.

# List: O(n) - must check each element
my_list = [2, 7, 11, 15]
if 7 in my_list:  # Checks: 2? no. 7? yes! (2 checks)
    print("Found")

# Dictionary: O(1) - instant lookup
my_dict = {2: 'a', 7: 'b', 11: 'c', 15: 'd'}
if 7 in my_dict:  # Goes directly to location (1 check)
    print("Found")

Practical Example: Two Sum Problem

Problem: Find two numbers that add up to a target.

Slow approach (nested loops - O(n²)):

nums = [2, 7, 11, 15]
target = 9

for i in range(len(nums)):
    for j in range(i + 1, len(nums)):
        if nums[i] + nums[j] == target:
            print([i, j])  # [0, 1]

Fast approach (dictionary - O(n)):

nums = [2, 7, 11, 15]
target = 9
seen = {}

for i, num in enumerate(nums):
    complement = target - num
    if complement in seen:
        print([seen[complement], i])  # [0, 1]
    else:
        seen[num] = i

Why it's faster:

• We loop once through the array
• For each number, we check if its complement exists (O(1) lookup)
• Total: O(n) instead of O(n²)

Trace through:

i=0, num=2: complement=7, not in seen, add {2: 0}
i=1, num=7: complement=2, IS in seen at index 0, return [0, 1]

Exercises

Exercise 1: Create a dictionary of 5 countries and their capitals. Print each country and its capital.

Exercise 2: Write a program that counts how many times each character appears in a string.

Exercise 3: Given a list of numbers, create a dictionary where keys are numbers and values are their squares.

Exercise 4: Create a program that stores product names and prices. Let the user look up prices by product name.

Exercise 5: Given a 5×5 list of numbers, count how many times each number appears and print the three most common.

Exercise 6: DNA pattern matching - given a list of DNA sequences and a pattern with wildcards (*), find matching sequences:

sequences = ['ATGCATGC', 'ATGGATGC', 'TTGCATGC']
pattern = 'ATG*ATGC'  # * matches any character
# Should match: 'ATGCATGC', 'ATGGATGC'

# Exercise 1
capitals = {'France': 'Paris', 'Japan': 'Tokyo', 'Italy': 'Rome', 
            'Egypt': 'Cairo', 'Brazil': 'Brasilia'}
for country, capital in capitals.items():
    print(f"{country}: {capital}")

# Exercise 2
text = "hello world"
char_count = {}
for char in text:
    char_count[char] = char_count.get(char, 0) + 1
print(char_count)

# Exercise 3
numbers = [1, 2, 3, 4, 5]
squares = {n: n**2 for n in numbers}
print(squares)  # {1: 1, 2: 4, 3: 9, 4: 16, 5: 25}

# Exercise 4
products = {}
while True:
    name = input("Product name (or 'done'): ")
    if name == 'done':
        break
    price = float(input("Price: "))
    products[name] = price

while True:
    lookup = input("Look up product (or 'quit'): ")
    if lookup == 'quit':
        break
    print(products.get(lookup, "Product not found"))

# Exercise 5
import random
grid = [[random.randint(1, 10) for _ in range(5)] for _ in range(5)]
counts = {}
for row in grid:
    for num in row:
        counts[num] = counts.get(num, 0) + 1
# Sort by count and get top 3
top3 = sorted(counts.items(), key=lambda x: x[1], reverse=True)[:3]
print("Top 3:", top3)

# Exercise 6
sequences = ['ATGCATGC', 'ATGGATGC', 'TTGCATGC']
pattern = 'ATG*ATGC'

for seq in sequences:
    match = True
    for i, char in enumerate(pattern):
        if char != '*' and char != seq[i]:
            match = False
            break
    if match:
        print(f"Match: {seq}")

Summary

Key takeaways:

• Dictionaries are fast for lookups (O(1))
• Use .get() for safe access with default values
• Loop with .items() to get both keys and values
• Python 3.7+ maintains insertion order
• Perfect for counting, caching, and mapping data

Regular Expressions in Python

Examples:

• Find all email addresses in a document
• Validate phone numbers
• Extract gene IDs from biological data
• Find DNA/RNA sequence patterns
• Clean messy text data

Getting Started

Import the Module

import re

Why Raw Strings Matter

# Normal string - \n becomes a newline
print("Hello\nWorld")
# Output:
# Hello
# World

# Raw string - \n stays as literal characters
print(r"Hello\nWorld")
# Output: Hello\nWorld

In regex, backslashes are special! Raw strings prevent confusion:

# ❌ Confusing without raw string
pattern = "\\d+"

# ✅ Clean with raw string
pattern = r"\d+"

Level 1: Literal Matching

The simplest regex matches exact text.

import re

dna = "ATGCGATCG"

# Search for exact text "ATG"
if re.search(r"ATG", dna):
    print("Found ATG!")

Your First Function: re.search()

match = re.search(r"ATG", "ATGCCC")
if match:
    print("Found:", match.group())    # Found: ATG
    print("Position:", match.start())  # Position: 0

Practice

Level 2: The Dot . - Match Any Character

The dot . matches any single character (except newline).

# Find "A" + any character + "G"
dna = "ATGCGATCG"
matches = re.findall(r"A.G", dna)
print(matches)  # ['ATG', 'ACG']

New Function: re.findall()

text = "cat bat rat"
print(re.findall(r".at", text))  # ['cat', 'bat', 'rat']

Practice

Level 3: Character Classes [ ]

Square brackets let you specify which characters to match.

# Match any nucleotide (A, T, G, or C)
dna = "ATGCXYZ"
nucleotides = re.findall(r"[ATGC]", dna)
print(nucleotides)  # ['A', 'T', 'G', 'C']

Character Ranges

Use - for ranges:

re.findall(r"[0-9]", "Room 123")      # ['1', '2', '3']
re.findall(r"[a-z]", "Hello")         # ['e', 'l', 'l', 'o']
re.findall(r"[A-Z]", "Hello")         # ['H']
re.findall(r"[A-Za-z]", "Hello123")   # ['H', 'e', 'l', 'l', 'o']

Negation with ^

^ inside brackets means "NOT these characters":

# Match anything that's NOT a nucleotide
dna = "ATGC-X123"
non_nucleotides = re.findall(r"[^ATGC]", dna)
print(non_nucleotides)  # ['-', 'X', '1', '2', '3']

Practice

Level 4: Quantifiers - Repeating Patterns

Quantifiers specify how many times a pattern repeats.

Examples

# Find sequences of 2+ C's
dna = "ATGCCCAAAGGG"
print(re.findall(r"C+", dna))       # ['CCC']
print(re.findall(r"C{2,}", dna))    # ['CCC']

# Find all digit groups
text = "Call 123 or 4567"
print(re.findall(r"\d+", text))     # ['123', '4567']

# Optional minus sign
print(re.findall(r"-?\d+", "123 -456 789"))  # ['123', '-456', '789']

Combining with Character Classes

# Find all 3-letter codons
dna = "ATGCCCAAATTT"
codons = re.findall(r"[ATGC]{3}", dna)
print(codons)  # ['ATG', 'CCC', 'AAA', 'TTT']

Practice

Level 5: Escaping Special Characters

Special characters like . * + ? [ ] ( ) have special meanings. To match them literally, escape with \.

# ❌ Wrong - dot matches ANY character
text = "file.txt and fileXtxt"
print(re.findall(r"file.txt", text))  # ['file.txt', 'fileXtxt']

# ✅ Correct - escaped dot matches only literal dot
print(re.findall(r"file\.txt", text))  # ['file.txt']

Common Examples

re.search(r"\$100", "$100")           # Literal dollar sign
re.search(r"What\?", "What?")         # Literal question mark
re.search(r"C\+\+", "C++")            # Literal plus signs
re.search(r"\(test\)", "(test)")      # Literal parentheses

Practice

Level 6: Predefined Shortcuts

Python provides shortcuts for common character types.

Examples

# Find all digits
text = "Room 123, Floor 4"
print(re.findall(r"\d+", text))  # ['123', '4']

# Find all words
sentence = "DNA_seq-123 test"
print(re.findall(r"\w+", sentence))  # ['DNA_seq', '123', 'test']

# Split on whitespace
data = "ATG  CCC\tAAA"
print(re.split(r"\s+", data))  # ['ATG', 'CCC', 'AAA']

Practice

Level 7: Anchors - Position Matching

Anchors match positions, not characters.

Examples

dna = "ATGCCCATG"

# Match only at start
print(re.search(r"^ATG", dna))   # Matches!
print(re.search(r"^CCC", dna))   # None

# Match only at end
print(re.search(r"ATG$", dna))   # Matches!
print(re.search(r"CCC$", dna))   # None

# Word boundaries - whole words only
text = "The cat concatenated strings"
print(re.findall(r"\bcat\b", text))  # ['cat'] - only the word
print(re.findall(r"cat", text))      # ['cat', 'cat'] - both

Practice

Level 8: Alternation - OR Operator |

The pipe | means "match this OR that".

# Match either ATG or AUG
dna = "ATG is DNA, AUG is RNA"
print(re.findall(r"ATG|AUG", dna))  # ['ATG', 'AUG']

# Match stop codons
rna = "AUGCCCUAAUAGUGA"
print(re.findall(r"UAA|UAG|UGA", rna))  # ['UAA', 'UAG', 'UGA']

Practice

Level 9: Groups and Capturing ( )

Parentheses create groups you can extract separately.

# Extract parts of an email
email = "user@example.com"
match = re.search(r"(\w+)@(\w+)\.(\w+)", email)
if match:
    print("Username:", match.group(1))   # user
    print("Domain:", match.group(2))     # example
    print("TLD:", match.group(3))        # com
    print("Full:", match.group(0))       # user@example.com

Named Groups

Use (?P<name>...) for readable names:

gene_id = "NM_001234"
match = re.search(r"(?P<prefix>[A-Z]+)_(?P<number>\d+)", gene_id)
if match:
    print(match.group('prefix'))  # NM
    print(match.group('number'))  # 001234

Practice

Level 10: More Useful Functions

re.sub() - Find and Replace

# Mask stop codons
dna = "ATGTAACCC"
masked = re.sub(r"TAA|TAG|TGA", "XXX", dna)
print(masked)  # ATGXXXCCC

# Clean multiple spaces
text = "too    many     spaces"
clean = re.sub(r"\s+", " ", text)
print(clean)  # "too many spaces"

re.compile() - Reusable Patterns

# Compile once, use many times (more efficient!)
pattern = re.compile(r"ATG")

for seq in ["ATGCCC", "TTTAAA", "GCGCGC"]:
    if pattern.search(seq):
        print(f"{seq} contains ATG")

Practice

Biological Examples

Validate DNA Sequences

def is_valid_dna(sequence):
    """Check if sequence contains only A, T, G, C"""
    return bool(re.match(r"^[ATGC]+$", sequence))

print(is_valid_dna("ATGCCC"))  # True
print(is_valid_dna("ATGXCC"))  # False

Find Restriction Sites

def find_ecori(dna):
    """Find EcoRI recognition sites (GAATTC)"""
    matches = re.finditer(r"GAATTC", dna)
    return [(m.start(), m.group()) for m in matches]

dna = "ATGGAATTCCCCGAATTC"
print(find_ecori(dna))  # [(3, 'GAATTC'), (12, 'GAATTC')]

Count Codons

def count_codons(dna):
    """Split DNA into codons (groups of 3)"""
    return re.findall(r"[ATGC]{3}", dna)

dna = "ATGCCCAAATTT"
print(count_codons(dna))  # ['ATG', 'CCC', 'AAA', 'TTT']

Extract Gene IDs

def extract_gene_ids(text):
    """Extract gene IDs like NM_123456"""
    return re.findall(r"[A-Z]{2}_\d+", text)

text = "Genes NM_001234 and XM_567890 are important"
print(extract_gene_ids(text))  # ['NM_001234', 'XM_567890']

Quick Reference

Key Functions Summary

Resources

• Regex 101 Interactive Tutorial
• Regex Tester
• Python re Documentation

Object-Oriented Programming V2

The shift:

• Before (imperative): Write instructions, use functions
• Now (OOP): Create objects that contain both data AND behavior

You've Been Using OOP All Along

Plot twist: every data type in Python is already a class.

# Lists are objects
my_list = [1, 2, 3]
my_list.append(4)      # Method call
my_list.reverse()      # Another method

# Strings are objects
name = "hello"
name.upper()           # Method call

# Even integers are objects
x = 5
x.__add__(3)           # Same as x + 3

Level 1: Your First Class

The Syntax

class ClassName:
    # stuff goes here
    pass

A Simple Counter

Let's build step by step.

Step 1: Empty class

class Counter:
    pass

Step 2: The constructor

class Counter:
    def __init__(self, value):
        self.val = value

Step 3: A method

class Counter:
    def __init__(self, value):
        self.val = value
    
    def tick(self):
        self.val = self.val + 1

Step 4: More methods

class Counter:
    def __init__(self, value):
        self.val = value
    
    def tick(self):
        self.val = self.val + 1
    
    def reset(self):
        self.val = 0
    
    def value(self):
        return self.val

Step 5: Use it

c1 = Counter(0)
c2 = Counter(3)

c1.tick()
c2.tick()

print(c1.value())    # 1
print(c2.value())    # 4

Level 2: Understanding the Pieces

The Constructor: __init__

def __init__(self, value):
    self.val = value

When you write Counter(5):

1. Python creates a new Counter object
2. Calls __init__ with value = 5
3. Returns the object

The self Parameter

self = "this object I'm working on"

def tick(self):
    self.val = self.val + 1

• self.val means "the val that belongs to THIS object"
• Each object has its own copy of self.val

c1 = Counter(0)
c2 = Counter(100)

c1.tick()           # c1.val becomes 1
print(c2.value())   # Still 100, different object

# Defining: include self
def tick(self):
    ...

# Calling: don't include self
c1.tick()    # NOT c1.tick(c1)

Instance Variables

Variables attached to self are instance variables — each object gets its own copy.

def __init__(self, value):
    self.val = value      # Instance variable
    self.count = 0        # Another one

Level 3: Special Methods (Magic Methods)

Python has special method names that enable built-in behaviors.

__str__ — For print()

class Counter:
    def __init__(self, value):
        self.val = value
    
    def __str__(self):
        return f"Counter: {self.val}"

c = Counter(5)
print(c)    # Counter: 5

Without __str__, you'd get something ugly like <__main__.Counter object at 0x7f...>

__add__ — For the + operator

class Counter:
    def __init__(self, value):
        self.val = value
    
    def __add__(self, other):
        return Counter(self.val + other.val)

c1 = Counter(3)
c2 = Counter(7)
c3 = c1 + c2        # Calls c1.__add__(c2)
print(c3.val)       # 10

__len__ — For len()

def __len__(self):
    return self.val

c = Counter(5)
print(len(c))    # 5

__getitem__ — For indexing [ ]

def __getitem__(self, index):
    return something[index]

Level 4: Encapsulation

Bad (direct access):

c = Counter(5)
c.val = -100       # Directly messing with internal data

Good (through methods):

c = Counter(5)
c.reset()          # Using the provided interface

Level 5: Designing Classes

When creating a class, think about:

Example: Student

• Class: Student
• Data: name, age, grades
• Behavior: add_grade(), average(), pass_course()

Full Example: Card Deck

Let's see a more complex class.

Step 1: Constructor — Create all cards

class Deck:
    def __init__(self):
        self.cards = []
        for num in range(1, 11):
            for suit in ["Clubs", "Spades", "Hearts", "Diamonds"]:
                self.cards.append((num, suit))

Step 2: Shuffle method

from random import randint

class Deck:
    def __init__(self):
        self.cards = []
        for num in range(1, 11):
            for suit in ["Clubs", "Spades", "Hearts", "Diamonds"]:
                self.cards.append((num, suit))
    
    def shuffle(self):
        for i in range(200):
            x = randint(0, len(self.cards) - 1)
            y = randint(0, len(self.cards) - 1)
            # Swap
            self.cards[x], self.cards[y] = self.cards[y], self.cards[x]

Step 3: Special methods

def __len__(self):
    return len(self.cards)

def __getitem__(self, i):
    return self.cards[i]

def __str__(self):
    return f"I am a {len(self)} card deck"

Step 4: Using it

deck = Deck()
print(deck)              # I am a 40 card deck
print(deck[0])           # (1, 'Clubs')

deck.shuffle()
print(deck[0])           # Something random now

Complete Counter Example

Putting it all together:

class Counter:
    def __init__(self, value):
        self.val = value
    
    def tick(self):
        self.val = self.val + 1
    
    def reset(self):
        self.val = 0
    
    def value(self):
        return self.val
    
    def __str__(self):
        return f"Counter: {self.val}"
    
    def __add__(self, other):
        return Counter(self.val + other.val)

c1 = Counter(0)
c2 = Counter(3)

c1.tick()
c2.tick()

c3 = c1 + c2

print(c1.value())    # 1
print(c2)            # Counter: 4
print(c3)            # Counter: 5

Quick Reference

OOP: Extra Practice

Exercise 1: Clock

Build a clock step by step.

Part A: Basic structure

Create a Clock class with:

• hours, minutes, seconds (all start at 0, or passed to constructor)
• A tick() method that adds 1 second

Part B: Handle overflow

Make tick() handle:

• 60 seconds → 1 minute
• 60 minutes → 1 hour
• 24 hours → back to 0

Part C: Display

Add __str__ to show time as "HH:MM:SS" (with leading zeros).

c = Clock(23, 59, 59)
print(c)        # 23:59:59
c.tick()
print(c)        # 00:00:00

Part D: Add seconds

Add __add__ to add an integer number of seconds:

c = Clock(10, 30, 0)
c2 = c + 90     # Add 90 seconds
print(c2)       # 10:31:30

Exercise 2: Fraction

Create a Fraction class for exact arithmetic (no floating point nonsense).

Part A: Constructor and display

f = Fraction(1, 2)
print(f)        # 1/2

Part B: Simplify automatically

Use math.gcd to always store fractions in simplest form:

f = Fraction(4, 8)
print(f)        # 1/2 (not 4/8)

Part C: Arithmetic

Add these special methods:

• __add__ → Fraction(1,2) + Fraction(1,3) = Fraction(5,6)
• __sub__ → subtraction
• __mul__ → multiplication
• __eq__ → Fraction(1,2) == Fraction(2,4) → True

Part D: Test these expressions

# Expression 1
f1 = Fraction(1, 4)
f2 = Fraction(1, 6)
f3 = Fraction(3, 2)
result = f1 + f2 * f3
print(result)           # Should be 1/2

# Expression 2
f4 = Fraction(1, 4)
f5 = Fraction(1, 4)
f6 = Fraction(1, 2)
print(f4 + f5 == f6)    # Should be True

Exercise 3: Calculator

A calculator that remembers its state.

Part A: Basic operations

class Calculator:
    # value starts at 0
    # add(x) → adds x to value
    # subtract(x) → subtracts x
    # multiply(x) → multiplies
    # divide(x) → divides
    # clear() → resets to 0
    # result() → returns current value

calc = Calculator()
calc.add(10)
calc.multiply(2)
calc.subtract(5)
print(calc.result())    # 15

Part B: Chain operations

Make methods return self so you can chain:

calc = Calculator()
calc.add(10).multiply(2).subtract(5)
print(calc.result())    # 15

Part C: Memory

Add:

• memory_store() → saves current value
• memory_recall() → adds stored value to current
• memory_clear() → clears memory

Exercise 4: Playlist

Part A: Song class

class Song:
    # title, artist, duration (in seconds)
    # __str__ returns "Artist - Title (M:SS)"

s = Song("Bohemian Rhapsody", "Queen", 354)
print(s)    # Queen - Bohemian Rhapsody (5:54)

Part B: Playlist class

class Playlist:
    # name
    # songs (list)
    # add_song(song)
    # total_duration() → returns total seconds
    # __len__ → number of songs
    # __getitem__ → access by index
    # __str__ → shows playlist name and song count

p = Playlist("Road Trip")
p.add_song(Song("Song A", "Artist 1", 180))
p.add_song(Song("Song B", "Artist 2", 240))

print(len(p))              # 2
print(p[0])                # Artist 1 - Song A (3:00)
print(p.total_duration())  # 420

Exercise 5: Quick Concepts

No code — just answer:

5.1: What's the difference between a class and an object?

5.2: Why do methods have self as first parameter?

5.3: What happens if you forget __str__ and try to print an object?

5.4: When would you use __eq__ instead of just comparing with ==?

5.5: What's encapsulation and why should you care?

Exercise 6: Debug This

class BankAccount:
    def __init__(self, balance):
        balance = balance
    
    def deposit(amount):
        balance += amount
    
    def __str__(self):
        return f"Balance: {self.balance}"

acc = BankAccount(100)
acc.deposit(50)
print(acc)

This crashes. Find all the bugs.

Object-Oriented Programming in Python

Object-Oriented Programming (OOP) is a way of organizing code by bundling related data and functions together into "objects". Instead of writing separate functions that work on data, you create objects that contain both the data and the functions that work with that data.

Why Learn OOP?

OOP helps you write code that is easier to understand, reuse, and maintain. It mirrors how we think about the real world - objects with properties and behaviors.

The four pillars of OOP:

1. Encapsulation - Bundle data and methods together
2. Abstraction - Hide complex implementation details
3. Inheritance - Create new classes based on existing ones
4. Polymorphism - Same interface, different implementations

Level 1: Understanding Classes and Objects

What is a Class?

A class is a blueprint or template for creating objects. Think of it like a cookie cutter - it defines the shape, but it's not the cookie itself.

# This is a class - a blueprint for dogs
class Dog:
    pass  # Empty for now

Naming Convention

Classes use PascalCase (UpperCamelCase):

class Dog:              # ✓ Good
class BankAccount:      # ✓ Good
class DataProcessor:    # ✓ Good

class my_class:         # ✗ Bad (snake_case)
class myClass:          # ✗ Bad (camelCase)

What is an Object (Instance)?

An object (or instance) is an actual "thing" created from the class blueprint. If the class is a cookie cutter, the object is the actual cookie.

class Dog:
    pass

# Creating objects (instances)
buddy = Dog()  # buddy is an object
max_dog = Dog()  # max_dog is another object

# Both are dogs, but they're separate objects
print(type(buddy))  # lass '__main__.Dog'>

Terminology:

• Dog is the class (blueprint)
• buddy and max_dog are instances or objects (actual things)
• We say: "buddy is an instance of Dog" or "buddy is a Dog object"

Level 2: Attributes - Giving Objects Data

Attributes are variables that store data inside an object. They represent the object's properties or state.

Instance Attributes

Instance attributes are unique to each object:

class Dog:
    def __init__(self, name, age):
        self.name = name  # Instance attribute
        self.age = age    # Instance attribute

# Create two different dogs
buddy = Dog("Buddy", 3)
max_dog = Dog("Max", 5)

# Each has its own attributes
print(buddy.name)    # "Buddy"
print(max_dog.name)  # "Max"
print(buddy.age)     # 3
print(max_dog.age)   # 5

Understanding __init__

__init__ is a special method called a constructor. It runs automatically when you create a new object.

class Dog:
    def __init__(self, name, age):
        print(f"Creating a dog named {name}!")
        self.name = name
        self.age = age

buddy = Dog("Buddy", 3)  
# Prints: "Creating a dog named Buddy!"

What __init__ does:

• Initializes (sets up) the new object's attributes
• Runs automatically when you call Dog(...)
• First parameter is always self

The double underscores (__init__) are called "dunder" (double-underscore). These mark special methods that Python recognizes for specific purposes.

Understanding self

self refers to the specific object you're working with:

class Dog:
    def __init__(self, name):
        self.name = name  # self.name means "THIS dog's name"

buddy = Dog("Buddy")
# When creating buddy, self refers to buddy
# So self.name = "Buddy" stores "Buddy" in buddy's name attribute

max_dog = Dog("Max")
# When creating max_dog, self refers to max_dog
# So self.name = "Max" stores "Max" in max_dog's name attribute

Important:

• self is just a naming convention (you could use another name, but don't!)
• Always include self as the first parameter in methods
• You don't pass self when calling methods - Python does it automatically

Class Attributes

Class attributes are shared by ALL objects of that class:

class Dog:
    species = "Canis familiaris"  # Class attribute (shared)
    
    def __init__(self, name):
        self.name = name  # Instance attribute (unique)

buddy = Dog("Buddy")
max_dog = Dog("Max")

print(buddy.species)   # "Canis familiaris"
print(max_dog.species) # "Canis familiaris" (same for both)
print(buddy.name)      # "Buddy" (different)
print(max_dog.name)    # "Max" (different)

Practice:

Exercise 1: Create a Cat class with name and color attributes

Exercise 2: Create two cat objects with different names and colors

Exercise 3: Create a Book class with title, author, and pages attributes

Exercise 4: Add a class attribute book_count to track how many books exist

Exercise 5: Create a Student class with name and grade attributes

# Exercise 1 & 2
class Cat:
    def __init__(self, name, color):
        self.name = name
        self.color = color

whiskers = Cat("Whiskers", "orange")
mittens = Cat("Mittens", "black")
print(whiskers.name, whiskers.color)  # Whiskers orange
print(mittens.name, mittens.color)    # Mittens black

# Exercise 3
class Book:
    def __init__(self, title, author, pages):
        self.title = title
        self.author = author
        self.pages = pages

book1 = Book("Python Basics", "John Doe", 300)
print(book1.title)  # Python Basics

# Exercise 4
class Book:
    book_count = 0  # Class attribute
    
    def __init__(self, title, author):
        self.title = title
        self.author = author
        Book.book_count += 1

book1 = Book("Book 1", "Author 1")
book2 = Book("Book 2", "Author 2")
print(Book.book_count)  # 2

# Exercise 5
class Student:
    def __init__(self, name, grade):
        self.name = name
        self.grade = grade

student = Student("Alice", "A")
print(student.name, student.grade)  # Alice A

Level 3: Methods - Giving Objects Behavior

Methods are functions defined inside a class. They define what objects can do.

Instance Methods

Instance methods operate on a specific object and can access its attributes:

class Dog:
    def __init__(self, name, age):
        self.name = name
        self.age = age
    
    def bark(self):  # Instance method
        return f"{self.name} says Woof!"
    
    def get_age_in_dog_years(self):
        return self.age * 7

buddy = Dog("Buddy", 3)
print(buddy.bark())                    # "Buddy says Woof!"
print(buddy.get_age_in_dog_years())    # 21

Key points:

• First parameter is always self
• Can access object's attributes using self.attribute_name
• Called using dot notation: object.method()

Methods Can Modify Attributes

Methods can both read and change an object's attributes:

class BankAccount:
    def __init__(self, balance):
        self.balance = balance
    
    def deposit(self, amount):
        self.balance += amount  # Modify the balance
        return self.balance
    
    def withdraw(self, amount):
        if amount <= self.balance:
            self.balance -= amount
            return self.balance
        else:
            return "Insufficient funds"
    
    def get_balance(self):
        return self.balance

account = BankAccount(100)
account.deposit(50)
print(account.get_balance())  # 150
account.withdraw(30)
print(account.get_balance())  # 120

Practice: Methods

Exercise 1: Add a meow() method to the Cat class

Exercise 2: Add a have_birthday() method to Dog that increases age by 1

Exercise 3: Create a Rectangle class with width, height, and methods area() and perimeter()

Exercise 4: Add a description() method to Book that returns a formatted string

Exercise 5: Create a Counter class with increment(), decrement(), and reset() methods

# Exercise 1
class Cat:
    def __init__(self, name):
        self.name = name
    
    def meow(self):
        return f"{self.name} says Meow!"

cat = Cat("Whiskers")
print(cat.meow())  # Whiskers says Meow!

# Exercise 2
class Dog:
    def __init__(self, name, age):
        self.name = name
        self.age = age
    
    def have_birthday(self):
        self.age += 1
        return f"{self.name} is now {self.age} years old!"

dog = Dog("Buddy", 3)
print(dog.have_birthday())  # Buddy is now 4 years old!

# Exercise 3
class Rectangle:
    def __init__(self, width, height):
        self.width = width
        self.height = height
    
    def area(self):
        return self.width * self.height
    
    def perimeter(self):
        return 2 * (self.width + self.height)

rect = Rectangle(5, 3)
print(rect.area())       # 15
print(rect.perimeter())  # 16

# Exercise 4
class Book:
    def __init__(self, title, author, pages):
        self.title = title
        self.author = author
        self.pages = pages
    
    def description(self):
        return f"'{self.title}' by {self.author}, {self.pages} pages"

book = Book("Python Basics", "John Doe", 300)
print(book.description())  # 'Python Basics' by John Doe, 300 pages

# Exercise 5
class Counter:
    def __init__(self):
        self.count = 0
    
    def increment(self):
        self.count += 1
    
    def decrement(self):
        self.count -= 1
    
    def reset(self):
        self.count = 0
    
    def get_count(self):
        return self.count

counter = Counter()
counter.increment()
counter.increment()
print(counter.get_count())  # 2
counter.decrement()
print(counter.get_count())  # 1
counter.reset()
print(counter.get_count())  # 0

Level 4: Inheritance - Reusing Code

Inheritance lets you create a new class based on an existing class. The new class inherits attributes and methods from the parent.

Why? Code reuse - don't repeat yourself!

Basic Inheritance

# Parent class (also called base class or superclass)
class Animal:
    def __init__(self, name):
        self.name = name
    
    def speak(self):
        return "Some sound"

# Child class (also called derived class or subclass)
class Dog(Animal):  # Dog inherits from Animal
    def speak(self):  # Override parent method
        return f"{self.name} says Woof!"

class Cat(Animal):
    def speak(self):
        return f"{self.name} says Meow!"

dog = Dog("Buddy")
cat = Cat("Whiskers")

print(dog.speak())  # "Buddy says Woof!"
print(cat.speak())  # "Whiskers says Meow!"

What happened:

• Dog and Cat inherit __init__ from Animal (no need to rewrite it!)
• Both override the speak method with their own version
• Each child gets all parent attributes and methods automatically

Extending Parent's __init__ with super()

Use super() to call the parent's __init__ and then add more:

class Animal:
    def __init__(self, name):
        self.name = name

class Dog(Animal):
    def __init__(self, name, breed):
        super().__init__(name)  # Call parent's __init__
        self.breed = breed      # Add new attribute
    
    def info(self):
        return f"{self.name} is a {self.breed}"

dog = Dog("Buddy", "Golden Retriever")
print(dog.info())  # "Buddy is a Golden Retriever"
print(dog.name)    # "Buddy" (inherited from Animal)

Method Overriding

Method overriding happens when a child class provides its own implementation of a parent's method:

class Animal:
    def speak(self):
        return "Some sound"
    
    def move(self):
        return "Moving"

class Fish(Animal):
    def move(self):  # Override
        return "Swimming"
    
    def speak(self):  # Override
        return "Blub"

class Bird(Animal):
    def move(self):  # Override
        return "Flying"
    # speak() not overridden, so uses parent's version

fish = Fish()
bird = Bird()

print(fish.move())   # "Swimming" (overridden)
print(fish.speak())  # "Blub" (overridden)
print(bird.move())   # "Flying" (overridden)
print(bird.speak())  # "Some sound" (inherited, not overridden)

Rule: When you call a method, Python uses the child's version if it exists, otherwise the parent's version.

Practice: Inheritance

Exercise 1: Create a Vehicle parent class with brand and year attributes

Exercise 2: Create Car and Motorcycle child classes that inherit from Vehicle

Exercise 3: Override a description() method in each child class

Exercise 4: Create an Employee parent class and a Manager child class with additional department attribute

Exercise 5: Create a Shape parent with color attribute, and Circle and Square children

# Exercise 1, 2, 3
class Vehicle:
    def __init__(self, brand, year):
        self.brand = brand
        self.year = year
    
    def description(self):
        return f"{self.year} {self.brand}"

class Car(Vehicle):
    def description(self):
        return f"{self.year} {self.brand} Car"

class Motorcycle(Vehicle):
    def description(self):
        return f"{self.year} {self.brand} Motorcycle"

car = Car("Toyota", 2020)
bike = Motorcycle("Harley", 2019)
print(car.description())   # 2020 Toyota Car
print(bike.description())  # 2019 Harley Motorcycle

# Exercise 4
class Employee:
    def __init__(self, name, salary):
        self.name = name
        self.salary = salary

class Manager(Employee):
    def __init__(self, name, salary, department):
        super().__init__(name, salary)
        self.department = department
    
    def info(self):
        return f"{self.name} manages {self.department}"

manager = Manager("Alice", 80000, "Sales")
print(manager.info())  # Alice manages Sales
print(manager.salary)  # 80000

# Exercise 5
class Shape:
    def __init__(self, color):
        self.color = color

class Circle(Shape):
    def __init__(self, color, radius):
        super().__init__(color)
        self.radius = radius
    
    def area(self):
        return 3.14159 * self.radius ** 2

class Square(Shape):
    def __init__(self, color, side):
        super().__init__(color)
        self.side = side
    
    def area(self):
        return self.side ** 2

circle = Circle("red", 5)
square = Square("blue", 4)
print(circle.area())   # 78.53975
print(circle.color)    # red
print(square.area())   # 16
print(square.color)    # blue

Level 5: Special Decorators for Methods

Decorators modify how methods behave. They're marked with @ symbol before the method.

@property - Methods as Attributes

Makes a method accessible like an attribute (no parentheses needed):

class Circle:
    def __init__(self, radius):
        self._radius = radius
    
    @property
    def radius(self):
        return self._radius
    
    @property
    def area(self):
        return 3.14159 * self._radius ** 2
    
    @property
    def circumference(self):
        return 2 * 3.14159 * self._radius

circle = Circle(5)
print(circle.radius)         # 5 (no parentheses!)
print(circle.area)           # 78.53975 (calculated on access)
print(circle.circumference)  # 31.4159

@staticmethod - Methods Without self

Static methods don't need access to the instance:

class Math:
    @staticmethod
    def add(x, y):
        return x + y
    
    @staticmethod
    def multiply(x, y):
        return x * y

# Call without creating an instance
print(Math.add(5, 3))       # 8
print(Math.multiply(4, 7))  # 28

@classmethod - Methods That Receive the Class

Class methods receive the class itself (not the instance):

class Dog:
    count = 0  # Class attribute
    
    def __init__(self, name):
        self.name = name
        Dog.count += 1
    
    @classmethod
    def get_count(cls):
        return f"There are {cls.count} dogs"
    
    @classmethod
    def create_default(cls):
        return cls("Default Dog")

dog1 = Dog("Buddy")
dog2 = Dog("Max")
print(Dog.get_count())  # "There are 2 dogs"

# Create a dog using class method
dog3 = Dog.create_default()
print(dog3.name)        # "Default Dog"
print(Dog.get_count())  # "There are 3 dogs"

Practice: Decorators

Exercise 1: Create a Temperature class with celsius property and fahrenheit property

Exercise 2: Add a static method is_freezing(celsius) to check if temperature is below 0

Exercise 3: Create a Person class with class method to count total people created

Exercise 4: Add a property age to calculate age from birth year

Exercise 5: Create utility class StringUtils with static methods for string operations

# Exercise 1
class Temperature:
    def __init__(self, celsius):
        self._celsius = celsius
    
    @property
    def celsius(self):
        return self._celsius
    
    @property
    def fahrenheit(self):
        return (self._celsius * 9/5) + 32

temp = Temperature(25)
print(temp.celsius)     # 25
print(temp.fahrenheit)  # 77.0

# Exercise 2
class Temperature:
    def __init__(self, celsius):
        self._celsius = celsius
    
    @property
    def celsius(self):
        return self._celsius
    
    @staticmethod
    def is_freezing(celsius):
        return celsius < 0

print(Temperature.is_freezing(-5))  # True
print(Temperature.is_freezing(10))  # False

# Exercise 3
class Person:
    count = 0
    
    def __init__(self, name):
        self.name = name
        Person.count += 1
    
    @classmethod
    def get_total_people(cls):
        return cls.count

p1 = Person("Alice")
p2 = Person("Bob")
print(Person.get_total_people())  # 2

# Exercise 4
class Person:
    def __init__(self, name, birth_year):
        self.name = name
        self.birth_year = birth_year
    
    @property
    def age(self):
        from datetime import datetime
        current_year = datetime.now().year
        return current_year - self.birth_year

person = Person("Alice", 1990)
print(person.age)  # Calculates current age

# Exercise 5
class StringUtils:
    @staticmethod
    def reverse(text):
        return text[::-1]
    
    @staticmethod
    def word_count(text):
        return len(text.split())
    
    @staticmethod
    def capitalize_words(text):
        return text.title()

print(StringUtils.reverse("hello"))           # "olleh"
print(StringUtils.word_count("hello world"))  # 2
print(StringUtils.capitalize_words("hello world"))  # "Hello World"

Level 6: Abstract Classes - Enforcing Rules

An abstract class is a class that cannot be instantiated directly. It exists only as a blueprint for other classes to inherit from.

Why? To enforce that child classes implement certain methods - it's a contract.

Creating Abstract Classes

Use the abc module (Abstract Base Classes):

from abc import ABC, abstractmethod

class Animal(ABC):  # Inherit from ABC
    def __init__(self, name):
        self.name = name
    
    @abstractmethod  # Must be implemented by children
    def speak(self):
        pass
    
    @abstractmethod
    def move(self):
        pass

# This will cause an error:
# animal = Animal("Generic")  # TypeError: Can't instantiate abstract class

class Dog(Animal):
    def speak(self):  # Must implement
        return f"{self.name} barks"
    
    def move(self):   # Must implement
        return f"{self.name} walks"

dog = Dog("Buddy")  # This works!
print(dog.speak())  # "Buddy barks"
print(dog.move())   # "Buddy walks"

Key points:

• Abstract classes inherit from ABC
• Use @abstractmethod for methods that must be implemented
• Child classes MUST implement all abstract methods
• Cannot create instances of abstract classes directly

Why Use Abstract Classes?

They enforce consistency across child classes:

from abc import ABC, abstractmethod

class Shape(ABC):
    @abstractmethod
    def area(self):
        pass
    
    @abstractmethod
    def perimeter(self):
        pass

class Rectangle(Shape):
    def __init__(self, width, height):
        self.width = width
        self.height = height
    
    def area(self):
        return self.width * self.height
    
    def perimeter(self):
        return 2 * (self.width + self.height)

class Circle(Shape):
    def __init__(self, radius):
        self.radius = radius
    
    def area(self):
        return 3.14159 * self.radius ** 2
    
    def perimeter(self):
        return 2 * 3.14159 * self.radius

# Both Rectangle and Circle MUST have area() and perimeter()
rect = Rectangle(5, 3)
circle = Circle(4)
print(rect.area())      # 15
print(circle.area())    # 50.26544

Practice: Abstract Classes

Exercise 1: Create an abstract Vehicle class with abstract method start_engine()

Exercise 2: Create abstract PaymentMethod class with abstract process_payment(amount) method

Exercise 3: Create concrete classes CreditCard and PayPal that inherit from PaymentMethod

Exercise 4: Create abstract Database class with abstract connect() and query() methods

Exercise 5: Create abstract FileProcessor with abstract read() and write() methods

# Exercise 1
from abc import ABC, abstractmethod

class Vehicle(ABC):
    @abstractmethod
    def start_engine(self):
        pass

class Car(Vehicle):
    def start_engine(self):
        return "Car engine started"

car = Car()
print(car.start_engine())  # Car engine started

# Exercise 2 & 3
class PaymentMethod(ABC):
    @abstractmethod
    def process_payment(self, amount):
        pass

class CreditCard(PaymentMethod):
    def __init__(self, card_number):
        self.card_number = card_number
    
    def process_payment(self, amount):
        return f"Charged ${amount} to card {self.card_number}"

class PayPal(PaymentMethod):
    def __init__(self, email):
        self.email = email
    
    def process_payment(self, amount):
        return f"Charged ${amount} to PayPal account {self.email}"

card = CreditCard("1234-5678")
paypal = PayPal("user@email.com")
print(card.process_payment(100))    # Charged $100 to card 1234-5678
print(paypal.process_payment(50))   # Charged $50 to PayPal account user@email.com

# Exercise 4
class Database(ABC):
    @abstractmethod
    def connect(self):
        pass
    
    @abstractmethod
    def query(self, sql):
        pass

class MySQL(Database):
    def connect(self):
        return "Connected to MySQL"
    
    def query(self, sql):
        return f"Executing MySQL query: {sql}"

db = MySQL()
print(db.connect())           # Connected to MySQL
print(db.query("SELECT *"))   # Executing MySQL query: SELECT *

# Exercise 5
class FileProcessor(ABC):
    @abstractmethod
    def read(self, filename):
        pass
    
    @abstractmethod
    def write(self, filename, data):
        pass

class TextFileProcessor(FileProcessor):
    def read(self, filename):
        return f"Reading text from {filename}"
    
    def write(self, filename, data):
        return f"Writing text to {filename}: {data}"

processor = TextFileProcessor()
print(processor.read("data.txt"))              # Reading text from data.txt
print(processor.write("out.txt", "Hello"))     # Writing text to out.txt: Hello

Level 7: Design Pattern - Template Method

The Template Method Pattern defines the skeleton of an algorithm in a parent class, but lets child classes implement specific steps.

from abc import ABC, abstractmethod

class DataProcessor(ABC):
    """Template for processing data"""
    
    def process(self):
        """Template method - defines the workflow"""
        data = self.load_data()
        cleaned = self.clean_data(data)
        result = self.analyze_data(cleaned)
        self.save_results(result)
    
    @abstractmethod
    def load_data(self):
        """Children must implement"""
        pass
    
    @abstractmethod
    def clean_data(self, data):
        """Children must implement"""
        pass
    
    @abstractmethod
    def analyze_data(self, data):
        """Children must implement"""
        pass
    
    def save_results(self, result):
        """Default implementation (can override)"""
        print(f"Saving: {result}")


class CSVProcessor(DataProcessor):
    def load_data(self):
        return "CSV data loaded"
    
    def clean_data(self, data):
        return f"{data} -> cleaned"
    
    def analyze_data(self, data):
        return f"{data} -> analyzed"


class JSONProcessor(DataProcessor):
    def load_data(self):
        return "JSON data loaded"
    
    def clean_data(self, data):
        return f"{data} -> cleaned differently"
    
    def analyze_data(self, data):
        return f"{data} -> analyzed differently"


# Usage
csv = CSVProcessor()
csv.process()
# Output: Saving: CSV data loaded -> cleaned -> analyzed

json = JSONProcessor()
json.process()
# Output: Saving: JSON data loaded -> cleaned differently -> analyzed differently

Benefits:

• Common workflow defined once in parent
• Each child implements specific steps differently
• Prevents code duplication
• Enforces consistent structure

Summary: Key Concepts

Classes and Objects

• Class = blueprint (use PascalCase)
• Object/Instance = actual thing created from class
• __init__ = constructor that runs when creating objects
• self = reference to the current object

Attributes and Methods

• Attributes = data (variables) stored in objects
• Instance attributes = unique to each object (defined in __init__)
• Class attributes = shared by all objects
• Methods = functions that define object behavior
• Access both using self.name inside the class

Inheritance

• Child class inherits from parent class
• Use super() to call parent's methods
• Method overriding = child replaces parent's method
• Promotes code reuse

Decorators

• @property = access method like an attribute
• @staticmethod = method without self, doesn't need instance
• @classmethod = receives class instead of instance
• @abstractmethod = marks methods that must be implemented

Abstract Classes

• Cannot be instantiated directly
• Use ABC and @abstractmethod
• Enforce that children implement specific methods
• Create contracts/interfaces

Design Patterns

• Template Method = define algorithm structure in parent, implement steps in children
• Promotes consistency and reduces duplication

Inheritance

Why bother?

• Reuse existing code
• Build specialized versions of general classes
• Organize related classes in hierarchies

The Basic Idea

Think about it:

• A Student is a Person
• A Student has everything a Person has (name, age, etc.)
• But a Student also has extra stuff (exams, courses, stress)

Instead of copy-pasting all the Person code into Student, we just say "Student inherits from Person" and add the extra bits.

Person (superclass / parent)
   ↓
Student (subclass / child)

Level 1: Creating a Subclass

The Syntax

Put the parent class name in parentheses:

class Student(Person):
    pass

That's it. Student now has everything Person has.

Let's Build It Step by Step

Step 1: The superclass

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age

Step 2: Add a method

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age
    
    def birthday(self):
        self.age += 1

Step 3: Add __str__

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age
    
    def birthday(self):
        self.age += 1
    
    def __str__(self):
        return f"{self.name}, age {self.age}"

Step 4: Create the subclass

class Student(Person):
    pass

Step 5: Test it

s = Student("Alice", 20)
print(s)           # Alice, age 20
s.birthday()
print(s)           # Alice, age 21

Level 2: Adding New Stuff to Subclasses

A subclass can have:

• Additional instance variables
• Additional methods
• Its own constructor

Adding a Method

class Student(Person):
    def study(self):
        print(f"{self.name} is studying...")

s = Student("Bob", 19)
s.study()      # Bob is studying...
s.birthday()   # Still works from Person!

Adding Instance Variables

Students have exams. Persons don't. Let's add that.

class Student(Person):
    def __init__(self, name, age):
        self.name = name
        self.age = age
        self.exams = []    # New!

Level 3: The super() Function

super() lets you call methods from the parent class. Use it to avoid code duplication.

Better Constructor

class Student(Person):
    def __init__(self, name, age):
        super().__init__(name, age)    # Call parent's __init__
        self.exams = []                 # Add our own stuff

Breaking it down:

super().__init__(name, age)

This says: "Hey parent class, run YOUR __init__ with these values."

Then we add the Student-specific stuff after.

Test It

s = Student("Charlie", 21)
print(s.name)      # Charlie (from Person)
print(s.exams)     # [] (from Student)

Level 4: Overriding Methods

If a subclass defines a method with the same name as the parent, it replaces (overrides) the parent's version.

Example: Override __str__

Parent version:

class Person:
    def __str__(self):
        return f"{self.name}, age {self.age}"

Child version (override):

class Student(Person):
    def __init__(self, name, age):
        super().__init__(name, age)
        self.exams = []
    
    def __str__(self):
        return f"Student: {self.name}, age {self.age}"

p = Person("Dan", 30)
s = Student("Eve", 20)

print(p)    # Dan, age 30
print(s)    # Student: Eve, age 20

Using super() in Overridden Methods

You can extend the parent's method instead of replacing it entirely:

class Student(Person):
    def __str__(self):
        base = super().__str__()           # Get parent's version
        return base + f", exams: {len(self.exams)}"

s = Student("Frank", 22)
print(s)    # Frank, age 22, exams: 0

Level 5: Inheritance vs Composition

The "is-a" Test

Ask yourself: "Is X a Y?"

When to Use Objects as Instance Variables

If X is NOT a Y, but X HAS a Y, use composition:

# A student HAS exams (not IS an exam)
class Student(Person):
    def __init__(self, name, age):
        super().__init__(name, age)
        self.exams = []    # List of Exam objects

# Exam is its own class, not a subclass
class Exam:
    def __init__(self, name, score, cfu):
        self.name = name
        self.score = score
        self.cfu = cfu

Level 6: Class Hierarchies

Subclasses can have their own subclasses. It's subclasses all the way down.

        Person
           ↓
        Student
           ↓
     ThesisStudent

class Person:
    pass

class Student(Person):
    pass

class ThesisStudent(Student):
    pass

A ThesisStudent inherits from Student, which inherits from Person.

The Secret: Everything Inherits from object

In Python, every class secretly inherits from object:

class Person:       # Actually: class Person(object)
    pass

That's why every class has methods like __str__ and __eq__ even if you don't define them (they're just not very useful by default).

Putting It Together: Complete Example

The Exam Class

class Exam:
    def __init__(self, name, score, cfu):
        self.name = name
        self.score = score
        self.cfu = cfu
    
    def __str__(self):
        return f"{self.name}: {self.score}/30 ({self.cfu} CFU)"

The Person Class

class Person:
    def __init__(self, name, age):
        self.name = name
        self.age = age
    
    def birthday(self):
        self.age += 1
    
    def __str__(self):
        return f"{self.name}, age {self.age}"

The Student Class

class Student(Person):
    def __init__(self, name, age):
        super().__init__(name, age)
        self.exams = []
    
    def pass_exam(self, exam):
        self.exams.append(exam)
    
    def __str__(self):
        base = super().__str__()
        if self.exams:
            exam_info = ", ".join(str(e) for e in self.exams)
            return f"{base}, exams: [{exam_info}]"
        return f"{base}, no exams yet"

Using It

# Create a student
s = Student("Grace", 20)
print(s)
# Grace, age 20, no exams yet

# Pass an exam
s.pass_exam(Exam("Python", 28, 6))
print(s)
# Grace, age 20, exams: [Python: 28/30 (6 CFU)]

# Pass another exam
s.pass_exam(Exam("Databases", 30, 9))
print(s)
# Grace, age 20, exams: [Python: 28/30 (6 CFU), Databases: 30/30 (9 CFU)]

# Have a birthday
s.birthday()
print(s)
# Grace, age 21, exams: [Python: 28/30 (6 CFU), Databases: 30/30 (9 CFU)]

Inheritance: Extra Practice

Exercise 1: Coffee Shop

You're building a coffee ordering system. ☕

Part A:

Create a Beverage class with:

• name and price
• __str__ that returns something like "Espresso: €2.50"

Part B:

Create a CustomBeverage subclass that:

• Has an extras list (e.g., ["oat milk", "extra shot"])
• Has an add_extra(extra_name, extra_price) method
• Each extra increases the total price
• Override __str__ to show the extras too

Test it:

drink = CustomBeverage("Latte", 3.00)
drink.add_extra("oat milk", 0.50)
drink.add_extra("vanilla syrup", 0.30)
print(drink)
# Latte: €3.80 (extras: oat milk, vanilla syrup)

Exercise 2: Shapes (Classic but Useful)

Part A:

Create a Shape class with:

• A name instance variable
• A method area() that returns 0 (base case)
• __str__ that returns "Shape: {name}, area: {area}"

Part B:

Create two subclasses:

Rectangle(Shape):

• Has width and height
• Override area() to return width * height

Circle(Shape):

• Has radius
• Override area() to return π * radius²

Part C:

Create a function (not a method!) that takes a list of shapes and returns the total area:

shapes = [Rectangle(4, 5), Circle(3), Rectangle(2, 2)]
print(total_area(shapes))  # Should work for any mix of shapes

Exercise 3: Game Characters

You're making an RPG. Because why not.

Part A:

Create a Character class with:

• name and health (default 100)
• take_damage(amount) that reduces health
• is_alive() that returns True if health > 0
• __str__ showing name and health

Part B:

Create a Warrior subclass:

• Has armor (default 10)
• Override take_damage so damage is reduced by armor first

Create a Mage subclass:

• Has mana (default 50)
• Has cast_spell(damage) that costs 10 mana and returns the damage (or 0 if no mana)

Test scenario:

w = Warrior("Ragnar")
m = Mage("Merlin")

w.take_damage(25)  # Should only take 15 damage (25 - 10 armor)
print(w)           # Ragnar: 85 HP

spell_damage = m.cast_spell(30)
print(m.mana)      # 40

Exercise 4: Quick Thinking

No code needed — just answer:

4.1: You have Animal and want to create Dog. Inheritance or instance variable?

4.2: You have Car and want to give it an Engine. Inheritance or instance variable?

4.3: What does super().__init__() do and when would you skip it?

4.4: If both Parent and Child have a method called greet(), which one runs when you call child_obj.greet()?

Exercise 5: Fix The Bug

This code has issues. Find and fix them:

class Vehicle:
    def __init__(self, brand):
        self.brand = brand
        self.fuel = 100
    
    def drive(self):
        self.fuel -= 10

class ElectricCar(Vehicle):
    def __init__(self, brand, battery):
        self.battery = battery
    
    def drive(self):
        self.battery -= 20

tesla = ElectricCar("Tesla", 100)
print(tesla.brand)  # 💥 Crashes! Why?

Quick Reference

Dynamic Programming

What is Dynamic Programming?

Dynamic Programming (DP) is an optimization technique that solves complex problems by breaking them down into simpler subproblems and storing their results to avoid redundant calculations.

The key idea: If you've already solved a subproblem, don't solve it again—just look up the answer!

Two fundamental principles:

1. Overlapping subproblems - the same smaller problems are solved multiple times
2. Optimal substructure - the optimal solution can be built from optimal solutions to subproblems

Why it matters: DP can transform exponentially slow algorithms into polynomial or even linear time algorithms by trading memory for speed.

Prerequisites: Why Dictionaries Are Perfect for DP

Before diving into dynamic programming, you should understand Python dictionaries. If you're not comfortable with dictionaries yet, review them first—they're the foundation of most DP solutions.

Quick dictionary essentials for DP:

# Creating and using dictionaries
cache = {}  # Empty dictionary

# Store results
cache[5] = 120
cache[6] = 720

# Check if we've seen this before
if 5 in cache:  # O(1) - instant lookup!
    print(cache[5])

# This is why dictionaries are perfect for DP!

Why dictionaries work for DP:

• O(1) lookup time - checking if a result exists is instant
• O(1) insertion time - storing a new result is instant
• Flexible keys - can store results for any input value
• Clear mapping - easy relationship between input (key) and result (value)

Now let's see DP in action with a classic example.

The Classic Example: Fibonacci

The Fibonacci sequence is perfect for understanding DP because it clearly shows the problem of redundant calculations.

The Problem: Naive Recursion

Fibonacci definition:

• F(0) = 0
• F(1) = 1
• F(n) = F(n-1) + F(n-2)

Naive recursive solution:

def fibonacci(n):
    if n <= 1:
        return n
    return fibonacci(n - 1) + fibonacci(n - 2)

print(fibonacci(10))  # 55
# Try fibonacci(40) - it takes forever!

Why Is This So Slow?

Look at the redundant calculations for fibonacci(5):

fibonacci(5)
├── fibonacci(4)
│   ├── fibonacci(3)
│   │   ├── fibonacci(2)
│   │   │   ├── fibonacci(1)  ← Calculated
│   │   │   └── fibonacci(0)  ← Calculated
│   │   └── fibonacci(1)      ← Calculated AGAIN
│   └── fibonacci(2)          ← Calculated AGAIN
│       ├── fibonacci(1)      ← Calculated AGAIN
│       └── fibonacci(0)      ← Calculated AGAIN
└── fibonacci(3)              ← Entire subtree calculated AGAIN
    ├── fibonacci(2)          ← Calculated AGAIN
    │   ├── fibonacci(1)      ← Calculated AGAIN
    │   └── fibonacci(0)      ← Calculated AGAIN
    └── fibonacci(1)          ← Calculated AGAIN

The numbers:

• fibonacci(1) is calculated 5 times
• fibonacci(2) is calculated 3 times
• fibonacci(3) is calculated 2 times

For fibonacci(40), you'd do 331,160,281 function calls. That's insane for a simple calculation!

Time complexity: O(2^n) - exponential! Each call spawns two more calls.

Dynamic Programming Solution: Memoization

Memoization = storing (caching) results we've already calculated using a dictionary.

# Dictionary to store computed results
memo = {}

def fibonacci_dp(n):
    # Check if we've already calculated this
    if n in memo:
        return memo[n]
    
    # Base cases
    if n <= 1:
        return n
    
    # Calculate, store, and return
    result = fibonacci_dp(n - 1) + fibonacci_dp(n - 2)
    memo[n] = result
    return result

# First call - calculates and stores results
print(fibonacci_dp(10))   # 55
print(memo)  # {2: 1, 3: 2, 4: 3, 5: 5, 6: 8, 7: 13, 8: 21, 9: 34, 10: 55}

# Subsequent calls - instant lookups!
print(fibonacci_dp(50))   # 12586269025 (instant!)
print(fibonacci_dp(100))  # Works perfectly, still instant!

How Memoization Works: Step-by-Step

Let's trace fibonacci_dp(5) with empty memo:

Call fibonacci_dp(5):
  5 not in memo
  Calculate: fibonacci_dp(4) + fibonacci_dp(3)
  
  Call fibonacci_dp(4):
    4 not in memo
    Calculate: fibonacci_dp(3) + fibonacci_dp(2)
    
    Call fibonacci_dp(3):
      3 not in memo
      Calculate: fibonacci_dp(2) + fibonacci_dp(1)
      
      Call fibonacci_dp(2):
        2 not in memo
        Calculate: fibonacci_dp(1) + fibonacci_dp(0)
        fibonacci_dp(1) = 1 (base case)
        fibonacci_dp(0) = 0 (base case)
        memo[2] = 1, return 1
      
      fibonacci_dp(1) = 1 (base case)
      memo[3] = 2, return 2
    
    Call fibonacci_dp(2):
      2 IS in memo! Return 1 immediately (no calculation!)
    
    memo[4] = 3, return 3
  
  Call fibonacci_dp(3):
    3 IS in memo! Return 2 immediately (no calculation!)
  
  memo[5] = 5, return 5

Final memo: {2: 1, 3: 2, 4: 3, 5: 5}

Notice: We only calculate each Fibonacci number once. All subsequent requests are instant dictionary lookups!

Time complexity: O(n) - we calculate each number from 0 to n exactly once
Space complexity: O(n) - we store n results in the dictionary

Comparison:

• Without DP: fibonacci(40) = 331,160,281 operations ⏰
• With DP: fibonacci(40) = 40 operations ⚡

That's over 8 million times faster!

Top-Down vs Bottom-Up Approaches

There are two main ways to implement DP:

Top-Down (Memoization) - What We Just Did

Start with the big problem and recursively break it down, storing results as you go.

memo = {}

def fib_topdown(n):
    if n in memo:
        return memo[n]
    if n <= 1:
        return n
    memo[n] = fib_topdown(n - 1) + fib_topdown(n - 2)
    return memo[n]

Pros:

• Intuitive if you think recursively
• Only calculates what's needed
• Easy to add memoization to existing recursive code

Cons:

• Uses recursion (stack space)
• Slightly slower due to function call overhead

Bottom-Up (Tabulation) - Build From Smallest

Start with the smallest subproblems and build up to the answer.

def fib_bottomup(n):
    if n <= 1:
        return n
    
    # Build table from bottom up
    dp = {0: 0, 1: 1}
    
    for i in range(2, n + 1):
        dp[i] = dp[i - 1] + dp[i - 2]
    
    return dp[n]

print(fib_bottomup(10))  # 55

Even more optimized (space-efficient):

def fib_optimized(n):
    if n <= 1:
        return n
    
    # We only need the last two values
    prev2, prev1 = 0, 1
    
    for i in range(2, n + 1):
        current = prev1 + prev2
        prev2, prev1 = prev1, current
    
    return prev1

print(fib_optimized(100))  # 354224848179261915075

Pros:

• No recursion (no stack overflow risk)
• Can optimize space usage (we did it above!)
• Often slightly faster

Cons:

• Less intuitive at first
• Calculates all subproblems even if not needed

When to Use Dynamic Programming

Use DP when you spot these characteristics:

1. Overlapping Subproblems

The same calculations are repeated many times.

Example: In Fibonacci, we calculate F(3) multiple times when computing F(5).

2. Optimal Substructure

The optimal solution to the problem contains optimal solutions to subproblems.

Example: The optimal path from A to C through B must include the optimal path from A to B.

3. You Can Define a Recurrence Relation

You can express the solution in terms of solutions to smaller instances.

Example: F(n) = F(n-1) + F(n-2)

Common DP Problem Patterns

1. Climbing Stairs

Problem: How many distinct ways can you climb n stairs if you can take 1 or 2 steps at a time?

def climbStairs(n):
    if n <= 2:
        return n
    
    memo = {1: 1, 2: 2}
    
    for i in range(3, n + 1):
        memo[i] = memo[i - 1] + memo[i - 2]
    
    return memo[n]

print(climbStairs(5))  # 8
# Ways: 1+1+1+1+1, 1+1+1+2, 1+1+2+1, 1+2+1+1, 2+1+1+1, 1+2+2, 2+1+2, 2+2+1

Key insight: This is actually Fibonacci in disguise! To reach step n, you either came from step n-1 (one step) or step n-2 (two steps).

2. Coin Change

Problem: Given coins of different denominations, find the minimum number of coins needed to make a target amount.

def coinChange(coins, amount):
    # dp[i] = minimum coins needed to make amount i
    dp = {0: 0}
    
    for i in range(1, amount + 1):
        min_coins = float('inf')
        
        # Try each coin
        for coin in coins:
            if i - coin >= 0 and i - coin in dp:
                min_coins = min(min_coins, dp[i - coin] + 1)
        
        if min_coins != float('inf'):
            dp[i] = min_coins
    
    return dp.get(amount, -1)

print(coinChange([1, 2, 5], 11))  # 3 (5 + 5 + 1)
print(coinChange([2], 3))          # -1 (impossible)

The DP Recipe: How to Solve DP Problems

1. Identify if it's a DP problem

   • Do you see overlapping subproblems?
   • Can you break it into smaller similar problems?

2. Define the state

   • What information do you need to solve each subproblem?
   • This becomes your dictionary key

3. Write the recurrence relation

   • How do you calculate dp[n] from smaller subproblems?
   • Example: F(n) = F(n-1) + F(n-2)

4. Identify base cases

   • What are the smallest subproblems you can solve directly?
   • Example: F(0) = 0, F(1) = 1

5. Implement and optimize

   • Start with top-down memoization (easier to write)
   • Optimize to bottom-up if needed
   • Consider space optimization

Common Mistakes to Avoid

1. Forgetting to Check the Cache

# Wrong - doesn't check memo first
def fib_wrong(n):
    if n <= 1:
        return n
    memo[n] = fib_wrong(n - 1) + fib_wrong(n - 2)  # Calculates every time!
    return memo[n]

# Correct - checks memo first
def fib_correct(n):
    if n in memo:  # Check first!
        return memo[n]
    if n <= 1:
        return n
    memo[n] = fib_correct(n - 1) + fib_correct(n - 2)
    return memo[n]

2. Not Storing the Result

# Wrong - calculates but doesn't store
def fib_wrong(n):
    if n in memo:
        return memo[n]
    if n <= 1:
        return n
    return fib_wrong(n - 1) + fib_wrong(n - 2)  # Doesn't store!

# Correct - stores before returning
def fib_correct(n):
    if n in memo:
        return memo[n]
    if n <= 1:
        return n
    memo[n] = fib_correct(n - 1) + fib_correct(n - 2)  # Store it!
    return memo[n]

3. Using Mutable Default Arguments

# Wrong - memo persists between calls!
def fib_wrong(n, memo={}):
    # ...

# Correct - create fresh memo or pass it explicitly
def fib_correct(n, memo=None):
    if memo is None:
        memo = {}
    # ...

Summary

Dynamic Programming is about:

• Recognizing overlapping subproblems
• Storing solutions to avoid recalculation
• Trading memory for speed

Key techniques:

• Top-down (memoization): Recursive + dictionary cache
• Bottom-up (tabulation): Iterative + build from smallest

When to use:

• Same subproblems solved repeatedly
• Optimal substructure exists
• Can define recurrence relation

The power of DP:

• Transforms exponential O(2^n) → linear O(n)
• Essential for many algorithmic problems
• Dictionaries make implementation clean and fast

Remember: Not every problem needs DP! Use it when you spot repeated calculations. Sometimes a simple loop or greedy algorithm is better.

Practice Problems to Try

1. House Robber - Maximum money you can rob from houses without robbing adjacent ones
2. Longest Common Subsequence - Find longest sequence common to two strings
3. Edit Distance - Minimum operations to convert one string to another
4. Maximum Subarray - Find contiguous subarray with largest sum
5. Unique Paths - Count paths in a grid from top-left to bottom-right

Each of these follows the same DP pattern we've learned. Try to identify the state, recurrence relation, and base cases!

Design Tic-Tac-Toe with Python

Project source: Hyperskill - Tic-Tac-Toe

Project Structure

This project is divided into multiple stages on Hyperskill, each with specific instructions and requirements. I'm sharing the final stage here, which integrates all previous components. The final stage instructions may seem brief as they build on earlier stages where the game logic was developed incrementally.

The complete input/output specifications can be found in the link above.

Sample Execution

---------
|       |
|       |
|       |
---------
3 1
---------
|       |
|       |
| X     |
---------
1 1
---------
| O     |
|       |
| X     |
---------
3 2
---------
| O     |
|       |
| X X   |
---------
0 0
Coordinates should be from 1 to 3!
1 2
---------
| O O   |
|       |
| X X   |
---------
3 3
---------
| O O   |
|       |
| X X X |
---------
X wins

Code

xo_arr = [[" "] * 3 for _ in range(3)]

def display_game(arr):
    row_one =  " ".join(xo_arr[0])
    row_two =  " ".join(xo_arr[1])
    row_three =  " ".join(xo_arr[2])

    print("---------")
    print(f"| {row_one} |")
    print(f"| {row_two} |")
    print(f"| {row_three} |")
    print("---------")


# This could be made in different(shorter) way, I think
# maybe make list of set all combinations for wining 
# and then check if it in or not 
def is_win(s):
    symbol_win =  xo_arr[0] == 3 * s
    symbol_win =  symbol_win or xo_arr[1] == 3 * s
    symbol_win =  symbol_win or xo_arr[2] == 3 * s

    symbol_win =  symbol_win or (xo_arr[0][0] == s and xo_arr[0][1] == s and xo_arr[0][2] == s)
    symbol_win =  symbol_win or (xo_arr[1][0] == s and xo_arr[1][1] == s and xo_arr[1][2] == s)
    symbol_win =  symbol_win or (xo_arr[2][0] == s and xo_arr[2][1] == s and xo_arr[2][2] == s)

    symbol_win =  symbol_win or (xo_arr[0][0] == s and xo_arr[1][1] == s and xo_arr[2][2] == s)
    symbol_win =  symbol_win or (xo_arr[0][2] == s and xo_arr[1][1] == s and xo_arr[2][0] == s)

    return symbol_win


symbol = "X"

display_game(xo_arr)


while True: 

    move = input()
    
    row_coordinate = move[0]
    column_coordinate = move[2]

    if not (row_coordinate.isdigit() and column_coordinate.isdigit()):
        print("You should enter numbers!")
        continue
    else:
        row_coordinate = int(row_coordinate)
        column_coordinate = int(column_coordinate)

    if not (1 <= row_coordinate <= 3 and 1 <= column_coordinate <= 3):
        print("Coordinates should be from 1 to 3!")
        continue

    elif xo_arr[row_coordinate - 1][column_coordinate - 1] == "X" or xo_arr[row_coordinate - 1][column_coordinate - 1] == "O":
        print("This cell is occupied! Choose another one!")
        continue

    xo_arr[row_coordinate - 1][column_coordinate - 1] = symbol

    if symbol == "X":
        symbol = "O"
    else:
        symbol = "X"

    display_game(xo_arr)

    o_win = is_win("O")
    x_win = is_win("X")


    if x_win:
        print("X wins")
        break

    elif o_win:
        print("O wins")
        break
    elif  " " not in xo_arr[0] and " " not in xo_arr[1] and " " not in xo_arr[2] :
        print("Draw")
        break

Multiplication Table

Write a multiplication table based on a maximum input value.

example:

> Please input number: 10
1    2    3    4    5    6    7    8    9    10  
2    4    6    8    10   12   14   16   18   20  
3    6    9    12   15   18   21   24   27   30  
4    8    12   16   20   24   28   32   36   40  
5    10   15   20   25   30   35   40   45   50  
6    12   18   24   30   36   42   48   54   60  
7    14   21   28   35   42   49   56   63   70  
8    16   24   32   40   48   56   64   72   80  
9    18   27   36   45   54   63   72   81   90  
10   20   30   40   50   60   70   80   90   100 

Implementation

This solution is dynamic because it depends on the number of digits in each result. If the maximum number in the table is 100, then the results can have:

three spaces → 1–9

two spaces → 10–99

one space → 100

So to align everything, you look at the biggest number in the table and check how many digits it has. You can do this mathematically (using tens) or simply by getting the length of the string of the number.

Then you add the right amount of spaces before each value to keep the table lined up.

num = int(input("Please input number: "))
max_spaces = len(str(num * num)) 
row = []

for i in range(1, num + 1):
    for j in range(1, num + 1):
        product = str(i * j)
        space =  " " * (max_spaces + 1 - len(product))
        row.append(product + space)
    
    print(*row)
    row = []

Sieve of Eratosthenes

This is an implementation of the Sieve of Eratosthenes.

You can find the full description of the algorithm on its Wikipedia page here.

Code

n = 120

consecutive_int  = [True for _ in range(2, n + 1)]

def mark_multiples(ci, p):
    for i in range(p * p, len(ci) + 2, p):
        ci[i - 2] = False
    return ci

def get_next_prime_notmarked(ci, p):
    for i in range(p + 1, len(ci) + 2):
        if ci[i - 2]:
            return i
    return - 1
            

next_prime = 2


while True:
    consecutive_int = mark_multiples(consecutive_int, next_prime)
    next_prime = get_next_prime_notmarked(consecutive_int, next_prime)
    if next_prime == -1:
        break

def convert_arr_nums(consecutive_int):
    num = ""
    for i in range(len(consecutive_int)):
        if consecutive_int[i]:
            num += str(i + 2) + " "
    return num
            

print(convert_arr_nums(consecutive_int))

Spiral Matrix

Difficulty: Medium
Source: LeetCode

Description

Given an m x n matrix, return all elements of the matrix in spiral order. The spiral traversal goes clockwise starting from the top-left corner: right → down → left → up, repeating inward until all elements are visited.

Code

# To be solved

Rotate Image

Difficulty: Medium
Source: LeetCode

Description

Given an n x n 2D matrix representing an image, rotate the image by 90 degrees clockwise.

Constraint: You must rotate the image in-place by modifying the input matrix directly. Do not allocate another 2D matrix.

Example

Input: matrix = [[1,2,3],[4,5,6],[7,8,9]]
Output: [[7,4,1],[8,5,2],[9,6,3]]

Code

# To be solved

Set Matrix Zeroes

Difficulty: Medium
Source: LeetCode

Description

Given an m x n integer matrix, if an element is 0, set its entire row and column to 0's.

Constraint: You must do it in place.

Example

Input: matrix = [[1,1,1],
                 [1,0,1],
                 [1,1,1]]
Output: [[1,0,1],
         [0,0,0],
         [1,0,1]]

Code

# To be solved

Two Pointers Intro

Reverse String

Difficulty: Easy
Source: LeetCode

Description

Write a function that reverse string in-place

Example

Input: s = ["h","e","l","l","o"]
Output: ["o","l","l","e","h"]

Code

# To be solved

Two Sum II - Input Array Is Sorted

Difficulty: Medium
Source: LeetCode

Description

You are given a 1-indexed integer array numbers that is sorted in non-decreasing order and an integer target.​

Your task is to return the 1-based indices of two different elements in numbers whose sum is exactly equal to target, with the guarantee that exactly one such pair exists

Please see full description in this link

Example

Example 1:

Input: numbers = [2, 7, 11, 15], target = 9​

Expected output: [1, 2]

Explanation: numbers[1] + numbers[2] = 2 + 7 = 9, so the correct indices are [1, 2].

Code

# To be solved

3sum

Difficulty: Medium
Source: LeetCode

Description

You are given an integer array nums, and the goal is to return all unique triplets [nums[i], nums[j], nums[k] such that each index is distinct and the sum of the three numbers is zero.​ The answer must not include duplicate triplets, even if the same values appear multiple times in the array.

Please see full description in this link

Example

Example 1:

Input: nums = [-1, 0, 1, 2, -1, -4]​

One valid output: [[-1, -1, 2], [-1, 0, 1]] (order of triplets or numbers within a triplet does not matter).

Code

# To be solved

Container With Most Water

Difficulty: Medium
Source: LeetCode

Description

You are given an array height where each element represents the height of a vertical line drawn at that index on the x-axis.​

Your goal is to pick two distinct lines such that, using the x-axis as the base, the container formed between these lines holds the maximum amount of water, and you must return that maximum water area

Please see full description in this link

Example

Example 1:

• Input: height = [1, 8, 6, 2, 5, 4, 8, 3, 7]​
• Output: 49
• Explanation (high level): The best container uses the line of height 8 and the line of height 7, which are far enough apart that the width and the shorter height together produce area 49.​

Code

# To be solved

Remove Duplicates from Sorted Array

Difficulty: Medium
Source: LeetCode

Description

You are given an integer array nums sorted in non-decreasing order, and you need to modify it in-place so that each distinct value appears only once in the prefix of the array.​ After the operation, you return an integer k representing how many unique values remain at the start of nums, and the first k positions should contain those unique values in their original relative order.​

Please see full description in this link

Example

Example 1:

• Input: nums = [1, 1, 2]​
• Output: k = 2 and nums’s first k elements become [1, 2, _] (the last position can hold any value)
• Explanation: The unique values are 1 and 2, so they occupy the first two positions and the function returns 2.​

Code

# To be solved

Move Zeroes

Difficulty: Medium
Source: LeetCode

Description

You are given an integer array nums and must move every 0 in the array to the end, without changing the relative order of the non-zero values.​ The rearrangement has to be performed directly on nums (in-place), and the overall extra space usage must remain O(1).

Please see full description in this link

Example

Example 1:

• Input: nums = [0, 1, 0, 3, 12]​
• Output (final state of nums): [1, 3, 12, 0, 0]
• Explanation: The non-zero elements 1, 3, 12 stay in the same relative order, and both zeros are moved to the end

Code

# To be solved

Valid Palindrome

Difficulty: Medium
Source: LeetCode

Description

You are given a string s consisting of printable ASCII characters, and the goal is to determine whether it forms a palindrome when considering only letters and digits and treating uppercase and lowercase as the same.​ After filtering out non-alphanumeric characters and converting all remaining characters to a single case, the cleaned string must read the same from left to right and right to left to be considered valid.​

Please see full description in this link

Example

Example 1:

• Input: s = "A man, a plan, a canal: Panama"​
• Output: True
• Explanation: After removing non-alphanumeric characters and lowering case, it becomes "amanaplanacanalpanama", which reads the same forwards and backwards.​

Code

# To be solved

Sliding Window Intro

Longest Substring Without Repeating Characters

• Difficulty: Medium
• Source: LeetCode 3 – Longest Substring Without Repeating Characters

Description

You are given a string s, and the goal is to determine the maximum length of any substring that has all unique characters, meaning no character appears more than once in that substring.

The substring must be contiguous within s (no reordering or skipping), and you only need to return the length of the longest such substring, not the substring itself.

Example

Example 1:

• Input: s = "abcabcbb"
• Output: 3
• Explanation: One longest substring without repeating characters is "abc", which has length 3.

Example 2:

• Input: s = "bbbbb"
• Output: 1
• Explanation: Every substring with unique characters is just "b", so the maximum length is 1.

Example 3:

• Input: s = "pwwkew"
• Output: 3
• Explanation: A valid longest substring is "wke" with length 3; note that "pwke" is not allowed because it is not contiguous.

You can test edge cases like s = "" (empty string) or s = " " (single space) to see how the result behaves.[6][8]

Code

# LeetCode 3: Longest Substring Without Repeating Characters
# Credit: Problem from LeetCode (see problem page for full statement and tests).

def lengthOfLongestSubstring(s: str) -> int:
    """
    Write your solution here.

    Requirements:
    - Consider contiguous substrings of s.
    - Within the chosen substring, all characters must be distinct.
    - Return the maximum length among all such substrings.
    """
    # To be solved
    raise NotImplementedError

Maximum Number of Vowels in a Substring of Given Length

Difficulty: Medium
Source: LeetCode

Description

Given a string s and an integer k, return the maximum number of vowel letters in any substring of s with length k.

Vowel letters in English are 'a', 'e', 'i', 'o', and 'u'.

Examples

Input: s = "abciiidef", k = 3
Output: 3
Explanation: The substring "iii" contains 3 vowel letters

Input: s = "aeiou", k = 2
Output: 2
Explanation: Any substring of length 2 contains 2 vowels

Input: s = "leetcode", k = 3
Output: 2
Explanation: "lee", "eet" and "ode" contain 2 vowels

Code

# To be solved

Climbing Stairs

Difficulty: Easy
Source: LeetCode

Description

You are climbing a staircase. It takes n steps to reach the top.

Each time you can either climb 1 or 2 steps. In how many distinct ways can you climb to the top?

Examples

Input: n = 2
Output: 2
Explanation: There are two ways to climb to the top:
1. 1 step + 1 step
2. 2 steps

Input: n = 3
Output: 3
Explanation: There are three ways to climb to the top:
1. 1 step + 1 step + 1 step
2. 1 step + 2 steps
3. 2 steps + 1 step

Code

# To be solved

Counting Bits

Difficulty: Easy
Source: LeetCode

Description

Given an integer n, return an array ans of length n + 1 such that for each i (0 <= i <= n), ans[i] is the number of 1's in the binary representation of i.

Example

Input: n = 2
Output: [0,1,1]
Explanation:
0 --> 0 (zero 1's)
1 --> 1 (one 1)
2 --> 10 (one 1)

Code

# To be solved

Decode Ways

Difficulty: Medium
Source: LeetCode

Description

Given a string s of digits, return the number of ways to decode it using the mapping:

"1" -> 'A', 
"2" -> 'B',
 ..., 
"26" -> 'Z'

A digit string can be decoded in multiple ways since some codes overlap (e.g., "12" can be "AB" or "L").

Rules:

• Valid codes are "1" to "26"
• Leading zeros are invalid (e.g., "06" is invalid, but "6" is valid)
• Return 0 if the string cannot be decoded

Examples

Input: s = "12"
Output: 2
Explanation: Can be decoded as "AB" (1, 2) or "L" (12)

Input: s = "11106"
Output: 2
Explanation: 
- "AAJF" with grouping (1, 1, 10, 6)
- "KJF" with grouping (11, 10, 6)
- (1, 11, 06) is invalid because "06" is not valid

Code

# To be solved

Maximal Square

Difficulty: Medium
Source: LeetCode

Description

Given an m x n binary matrix filled with 0's and 1's, find the largest square containing only 1's and return its area.

Example

Input: matrix = [
  ["1","0","1","0","0"],
  ["1","0","1","1","1"],
  ["1","1","1","1","1"],
  ["1","0","0","1","0"]
]
Output: 4
Explanation: The largest square of 1's has side length 2, so area = 2 × 2 = 4

Code

# To be solved

Word Break

Difficulty: Medium
Source: LeetCode

Description

Given a string s and a dictionary of strings wordDict, return true if s can be segmented into a space-separated sequence of one or more dictionary words.

Note: The same word in the dictionary may be reused multiple times in the segmentation.

Example

Input: s = "leetcode", wordDict = ["leet","code"]
Output: true
Explanation: "leetcode" can be segmented as "leet code"

Input: s = "applepenapple", wordDict = ["apple","pen"]
Output: true
Explanation: "applepenapple" can be segmented as "apple pen apple"
Note: "apple" is reused

Code

# To be solved

Longest Increasing Subsequence

Difficulty: Medium
Source: LeetCode

Description

Given an integer array nums, return the length of the longest strictly increasing subsequence.

A subsequence is derived by deleting some or no elements without changing the order of the remaining elements.

Example

Input: nums = [10,9,2,5,3,7,101,18]
Output: 4
Explanation: The longest increasing subsequence is [2,3,7,101], with length 4

Code

# To be solved

Subarray Sum Equals K

• Difficulty: Medium
• Source: LeetCode 560 – Subarray Sum Equals K

Problem credit: This note is for practicing the LeetCode problem “Subarray Sum Equals K”. For the full official statement, examples, and judge, see the LeetCode problem page.

Description

You are given an integer array nums and an integer k, and the task is to return the number of non-empty contiguous subarrays whose elements add up to k.

A subarray is defined as a sequence of one or more elements that appear consecutively in the original array, without reordering or skipping indices.

Example

Example 1:

• Input: nums = [1, 1, 1], k = 2
• Output: 2
• Explanation: The subarrays [1, 1] using indices [0, 1] and [1, 2] both sum to 2, so the answer is 2.

Example 2:

• Input: nums = [1, 2, 3], k = 3
• Output: 2
• Explanation: The subarrays [1, 2] and [3] each sum to 3, giving a total count of 2.

You can experiment with inputs that include negative numbers, such as [2, 2, -4, 1, 1, 2] and various k values, to see how multiple overlapping subarrays can share the same sum.

Code

# LeetCode 560: Subarray Sum Equals K
# Credit: Problem from LeetCode (see problem page for full statement and tests).

def subarraySum(nums: List[int], k: int) -> int:
    """
    Write your solution here.

    Requirements:
    - Count all non-empty contiguous subarrays whose sum is exactly k.
    - nums may contain positive, negative, and zero values.
    - Return the total number of such subarrays.
    """
    # To be solved
    raise NotImplementedError

Count Vowel Substrings of a String

Difficulty: Easy
Source: LeetCode

Description

Given a string word, return the number of vowel substrings in word.

A vowel substring is a contiguous substring that:

• Only consists of vowels ('a', 'e', 'i', 'o', 'u')
• Contains all five vowels at least once

Examples

Input: word = "aeiouu"
Output: 2
Explanation: The vowel substrings are "aeiou" and "aeiouu"

Input: word = "unicornarihan"
Output: 0
Explanation: Not all 5 vowels are present, so there are no vowel substrings

Code

# To be solved

Roman to Integer

The problem can be found here

Solution one

Let's think, simple solution for this problem, will be change the way that system work, in another word, instead of making minus, will make everything just sum.

class Solution:
    def romanToInt(self, s: str) -> int:
        roman = {
            "I": 1,
            "V": 5,
            "X": 10,
            "L": 50,
            "C": 100,
            "D": 500,
            "M": 1000
        }
        replace = {
            "IV": "IIII",
            "IX": "VIIII",
            "XL": "XXXX",
            "XC": "LXXXX",
            "CD": "CCCC",
            "CM": "DCCCC"
        }

        for k, v in replace.items(): 
            s = s.replace(k, v)
            
        return sum([roman[char] for char in s])

Solution two

Another way to think about this, is just if we say smaller number before bigger number, we should minus, otherwise, we should continue adding numbers.

class Solution:
    def romanToInt(self, s: str) -> int:
        roman = {
            "I": 1,
            "V": 5,
            "X": 10,
            "L": 50,
            "C": 100,
            "D": 500,
            "M": 1000
        }
        total = 0
        pre_value = 0

        for i in s:
            if pre_value < roman[i]:
                total += roman[i] - 2 * pre_value
            else:
                total += roman[i]
            
            pre_value = roman[i]
        
        return total

This solution in runtime beats 100%, but memory only 20% better

why I did this roman[i] - 2 * pre_value? because we need to minus the added value in the previous step.

Basic Calculator

Difficulty: Medium

Description

Given a string expression containing digits and operators (+, -, *, /), evaluate the expression and return the result.

Rules:

• Follow standard operator precedence (multiplication and division before addition and subtraction)
• Division should be integer division (truncate toward zero)
• No parentheses in the expression

Examples

Input: s = "3+2*2"
Output: 7
Explanation: Multiplication first: 3 + (2*2) = 3 + 4 = 7

Input: s = "4-8/2"
Output: 0
Explanation: Division first: 4 - (8/2) = 4 - 4 = 0

Input: s = "14/3*2"
Output: 8
Explanation: Left to right for same precedence: (14/3)*2 = 4*2 = 8

Code

# To be solved

String Metrics and Alignments

1. String Metrics and Alignments

What is a String Metric?

A string metric measures the distance or similarity between two text strings. We use these in bioinformatics to compare DNA, RNA, or protein sequences.

Edit Distance

Edit distances measure how many operations you need to transform one string into another. The operations can be:

Substitution: Replace one character with another

CAT → CUT
 |      
 A → U

Insertion/Deletion (Indel): Add or remove a character

MONKEY → MON-EY (deletion of K, or viewing it differently, insertion of gap)

Transposition: Swap adjacent characters

FORM → FROM

Each operation has a cost. The total cost defines the distance.

Distance vs Similarity

Distance metric: Range is [0, +∞)

• Score of 0 means identical strings
• Higher scores mean more different strings

Similarity metric: Range is (-∞, +∞)

• High positive score means similar strings
• Negative score means distant strings

In bioinformatics, we typically work with similarity because we want to find sequences that are alike.

2. Hamming Distance

The simplest edit distance. It only counts substitutions and requires strings of equal length.

Example

ACCCTCGCTAGATAAATAGATCTGATAG
||x||||||||||x|||||x||||x|||
ACTCTCGCTAGATGAATAGGTCTGTTAG

Count the mismatches (marked with x): positions 3, 14, 19, 25

Hamming Distance = 4

Converting to Similarity

For the example above:

• Total positions: 28
• Matches: 24
• Mismatches: 4

Similarity = +24 - 4 = 20

(We add +1 for each match, subtract 1 for each mismatch)

Implementation: AlnSeq Class (Basic)

We start by creating a class to hold two sequences and compute Hamming metrics.

Step 1: Define the class and constructor

class AlnSeq:
    def __init__(self, seq1, seq2):
        if len(seq1) != len(seq2):
            raise ValueError("Sequences must have equal length")
        
        self.seq1 = seq1
        self.seq2 = seq2

Step 2: Add Hamming distance method

    def compute_hamming_distance(self):
        self.distance = 0
        for i in range(len(self.seq1)):
            if self.seq1[i] != self.seq2[i]:
                self.distance += 1

        return self.distance

Step 3: Add Hamming similarity method

    def compute_hamming_similarity(self):
        if getattr(self, 'distance', None) == None:
            self.compute_hamming_distance()
            # +1 * numbers of matchs + (-1 * numbers of mismatchs)
            # It's possible simplify this math experssion, but left for readabilty
        return (len(self.seq1) - self.distance) - self.distance

Usage

aln = AlnSeq("ACCCTCGCTAG", "ACTCTCGCTAG")
print(aln.compute_hamming_distance())    # 1
print(aln.compute_hamming_similarity())  # 9

3. Biologically Relevant Substitution Matrices

Not all substitutions are equal in biology. Some amino acids or nucleotides are chemically similar, so swapping between them is "less bad" than swapping very different ones.

Transition/Transversion Matrix (for DNA)

Transitions are mutations between chemically similar bases:

• A ↔ G (both purines)
• T ↔ C (both pyrimidines)

Transversions are mutations between different types:

• A ↔ T, A ↔ C, G ↔ T, G ↔ C

      A    T    C    G
  A   2   -1   -1    0
  T  -1    2    0   -1
  C  -1    0    2   -1
  G   0   -1   -1    2

Example calculation:

AAAA
|XX/
ATCG

Score = Score(A,A) + Score(A,T) + Score(A,C) + Score(A,G) = 2 + (-1) + (-1) + 0 = 0

PAM Matrices (for Proteins)

PAM = Point Accepted Mutation

PAMn[i,j] gives the likelihood of residue i being replaced by residue j through evolutionary mutations over a time when n mutations occur per 100 residues.

Most commonly used: PAM250

BLOSUM Matrices (for Proteins)

BLOSUMn[i,j] is based on observed substitution frequencies in aligned protein sequences that share n% sequence identity.

Most commonly used: BLOSUM62

Implementation: SubstitutionMatrix Class

A class to store and access substitution matrix values.

Step 1: Define the class with matrix data

class SubstitutionMatrix:
    def __init__(self, matrix_dict):
        # matrix_dict is a nested dictionary
        # matrix_dict['A']['T'] gives score for A->T
        self.matrix = matrix_dict

Step 2: Implement getitem for easy indexing

This lets us use matrix['A', 'T'] syntax.

    def __getitem__(self, key):
        # key is a tuple like ('A', 'T')
        res1, res2 = key
        return self.matrix[res1][res2]

Step 3: Create the Transition/Transversion matrix

# Define the matrix as nested dictionaries
tt_matrix_data = {
    'A': {'A': 2, 'T': -1, 'C': -1, 'G': 0},
    'T': {'A': -1, 'T': 2, 'C': 0, 'G': -1},
    'C': {'A': -1, 'T': 0, 'C': 2, 'G': -1},
    'G': {'A': 0, 'T': -1, 'C': -1, 'G': 2}
}

tt_matrix = SubstitutionMatrix(tt_matrix_data)

Usage

print(tt_matrix['A', 'A'])   # 2  (match)
print(tt_matrix['A', 'G'])   # 0  (transition)
print(tt_matrix['A', 'T'])   # -1 (transversion)

Implementation: Extend AlnSeq with Matrix-Based Similarity

Add a method to compute similarity using any substitution matrix.

class AlnSeq:
    # ... previous methods ...
    
    def hamming_similarity_matrix(self, sub_matrix):
        if len(self.seq1) != len(self.seq2):
            raise ValueError("Sequences must have equal length")
        
        score = 0
        for i in range(len(self.seq1)):
            # Use the substitution matrix for scoring
            score += sub_matrix[self.seq1[i], self.seq2[i]]
        return score

Usage

aln = AlnSeq("AAAA", "ATCG")
print(aln.hamming_similarity_matrix(tt_matrix))  # 2 + (-1) + (-1) + 0 = 0

4. Levenshtein Distance

Unlike Hamming distance, Levenshtein allows insertions and deletions (indels), not just substitutions.

Example

KITTEN → SITTING

Step 1: K → S     (substitution)
Step 2: E → I     (substitution)  
Step 3: insert G  (insertion)

Levenshtein Distance = 3

Levenshtein vs Hamming

For strings of the same length: Hamming Distance ≥ Levenshtein Distance

FLAW    vs    LAWN

Hamming (no gaps allowed):
FLAW
XXXX
LAWN
Hamming Distance = 4

Levenshtein (gaps allowed):
-FLAW
 |||^
-LAWN
Levenshtein Distance = 2 (delete F, insert N)

This shows why we need sequence alignment.

5. Sequence Alignment

Each character of one sequence is matched with:

• A character in the other sequence, OR
• A gap (-)

After alignment, both gapped strings have the same length.

How Many Possible Alignments?

Simple case (no internal gaps): Just slide sequences past each other

--TCA    -TCA    TCA    TCA    TCA-    TCA--
TA---    TA--    TA-    -TA    --TA    ---TA

Number of alignments = len1 + len2 + 1

Complex case (internal gaps allowed): The number explodes

                    | 1           if len1 = 0
N(len1, len2) =     | 1           if len2 = 0
                    | N(len1-1, len2) + N(len1, len2-1) + N(len1-1, len2-1)  otherwise

Examples:

• N(3,3) = 63
• N(5,5) = 1,683
• N(9,9) = 1,462,563

Brute force is not feasible. We need a smarter approach.

Implementation: Counting Possible Alignments

A recursive function to calculate the number of possible alignments.

def count_alignments(len1, len2):
    # Base cases: if one sequence is empty,
    # only one alignment is possible (all gaps)
    if len1 == 0:
        return 1
    if len2 == 0:
        return 1
    
    # Recursive case: three choices at each position
    return (count_alignments(len1 - 1, len2) +      # gap in seq2
            count_alignments(len1, len2 - 1) +      # gap in seq1
            count_alignments(len1 - 1, len2 - 1))   # match/mismatch

Usage

print(count_alignments(3, 3))   # 63
print(count_alignments(5, 5))   # 1683
print(count_alignments(9, 9))   # 1462563

This function is slow because it recalculates the same subproblems many times. That is exactly why we need dynamic programming.

6. Dynamic Programming

An algorithmic paradigm that works when a problem has:

1. Optimal Substructure: The optimal solution depends on solutions to smaller sub-problems

2. Overlapping Subproblems: The same sub-problems are solved multiple times

Key idea: Store solutions to sub-problems and build up to the full solution.

7. Needleman-Wunsch Algorithm (Global Alignment)

Finds the best alignment across the entire length of both sequences.

Setup

Given:

• Sequence 1: TCA
• Sequence 2: TA
• Gap penalty: -2
• Substitution scores: Transition/Transversion matrix

Step 1: Initialize the Matrix

Create a matrix with sequence 1 across the top and sequence 2 down the side. Add a gap column/row at the start.

        -     T     C     A
   -    0    -2    -4    -6
   T   -2
   A   -4

First row: Each cell = previous + gap penalty (0, -2, -4, -6) First column: Same logic (0, -2, -4, -6)

Step 2: Fill the Matrix

For each cell (i,j), take the maximum of three options:

Cell[i,j] = max(
    Cell[i-1, j] + gap,           // come from above (gap in seq1)
    Cell[i, j-1] + gap,           // come from left (gap in seq2)  
    Cell[i-1, j-1] + match(i,j)   // come from diagonal (match/mismatch)
)

Filling cell (T,T):

max(-2 + (-2), 0 + match(T,T), -2 + (-2))
= max(-4, 0 + 2, -4)
= max(-4, 2, -4)
= 2

Complete filled matrix:

        -     T     C     A
   -    0    -2    -4    -6
   T   -2     2     0    -2
   A   -4     0     1     2

The bottom-right cell (2) is the optimal alignment score.

Step 3: Backtracking

Start from bottom-right. At each cell, go back the way you came:

• Diagonal: match characters
• Up: gap in sequence 1
• Left: gap in sequence 2

        -     T     C     A
   -    0 ←  -2 ←  -4 ←  -6
        ↑    ↖↑    ↖↑    ↖↑
   T   -2 ←   2  ←  0  ← -2
        ↑    ↖     ↖     ↖
   A   -4 ←   0  ←  1  ← [2]

Path from [2]: diagonal → diagonal → diagonal

Final Alignment:

TCA
|X|
T-A

Score = 2

8. Smith-Waterman Algorithm (Local Alignment)

Finds the best alignment between substrings of the sequences.

Differences from Needleman-Wunsch

1. Initialization: First row and column are all zeros (no penalty for starting/ending gaps)

2. Filling: Add a fourth option, zero:

Cell[i,j] = max(
    Cell[i-1, j] + gap,
    Cell[i, j-1] + gap,
    Cell[i-1, j-1] + match(i,j),
    0                              // NEW: discard negative paths
)

3. Backtracking: Start from the highest value in the matrix (not bottom-right). Stop when you hit zero.

Example

Sequences: TGA and GA, Gap penalty: -2

Initialization:

        -     T     G     A
   -    0     0     0     0
   G    0
   A    0

Filling cell (G,T):

max(0 + (-2), 0 + match(T,G), 0 + (-2), 0)
= max(-2, -1, -2, 0)
= 0

Complete matrix:

        -     T     G     A
   -    0     0     0     0
   G    0     0     2     0
   A    0     0     0    [4]

Backtracking from highest value [4]:

• (A,A): came from diagonal, match A-A
• (G,G): came from diagonal, match G-G
• (G,T): value is 0, STOP

Final Local Alignment:

GA
||
GA

Score = 4

The algorithm found the best matching substring, ignoring the T at the beginning.

Implementation: SeqPair Class (Needleman-Wunsch)

Step 1: Define the class

class SeqPair:
    def __init__(self, seq1, seq2, sub_matrix, gap_penalty=-2):
        self.seq1 = seq1
        self.seq2 = seq2
        self.sub_matrix = sub_matrix
        self.gap = gap_penalty

Step 2: Initialize the scoring matrix

    def _init_matrix(self, rows, cols):
        # Create matrix filled with zeros
        matrix = [[0] * cols for _ in range(rows)]
        return matrix

Step 3: Needleman-Wunsch matrix filling

    def needleman_wunsch(self):
        rows = len(self.seq2) + 1
        cols = len(self.seq1) + 1
        
        # Initialize score matrix
        score = self._init_matrix(rows, cols)
        
        # Initialize traceback matrix
        # 0 = diagonal, 1 = up, 2 = left
        trace = self._init_matrix(rows, cols)
        
        # Fill first row (gaps in seq2)
        for j in range(1, cols):
            score[0][j] = score[0][j-1] + self.gap
            trace[0][j] = 2  # came from left
        
        # Fill first column (gaps in seq1)
        for i in range(1, rows):
            score[i][0] = score[i-1][0] + self.gap
            trace[i][0] = 1  # came from up
        
        # Fill rest of matrix
        for i in range(1, rows):
            for j in range(1, cols):
                # Three options
                diag = score[i-1][j-1] + self.sub_matrix[self.seq1[j-1], self.seq2[i-1]]
                up = score[i-1][j] + self.gap
                left = score[i][j-1] + self.gap
                
                # Take maximum
                score[i][j] = max(diag, up, left)
                
                # Record which direction we came from
                if score[i][j] == diag:
                    trace[i][j] = 0
                elif score[i][j] == up:
                    trace[i][j] = 1
                else:
                    trace[i][j] = 2
        
        self.nw_score = score
        self.nw_trace = trace
        return score[rows-1][cols-1]  # final score

Step 4: Backtracking to get alignment

    def nw_traceback(self):
        aligned1 = ""
        aligned2 = ""
        
        # Start from bottom-right
        i = len(self.seq2)
        j = len(self.seq1)
        
        while i > 0 or j > 0:
            if i > 0 and j > 0 and self.nw_trace[i][j] == 0:
                # Diagonal: match/mismatch
                aligned1 = self.seq1[j-1] + aligned1
                aligned2 = self.seq2[i-1] + aligned2
                i -= 1
                j -= 1
            elif i > 0 and self.nw_trace[i][j] == 1:
                # Up: gap in seq1
                aligned1 = "-" + aligned1
                aligned2 = self.seq2[i-1] + aligned2
                i -= 1
            else:
                # Left: gap in seq2
                aligned1 = self.seq1[j-1] + aligned1
                aligned2 = "-" + aligned2
                j -= 1
        
        return aligned1, aligned2

Usage

seq_pair = SeqPair("TCA", "TA", tt_matrix, gap_penalty=-2)
score = seq_pair.needleman_wunsch()
print(f"Score: {score}")  # Score: 2

aln1, aln2 = seq_pair.nw_traceback()
print(aln1)  # TCA
print(aln2)  # T-A

Implementation: Smith-Waterman (Extend SeqPair)

Add local alignment method

    def smith_waterman(self):
        rows = len(self.seq2) + 1
        cols = len(self.seq1) + 1
        
        score = self._init_matrix(rows, cols)
        trace = self._init_matrix(rows, cols)
        
        # First row and column stay 0 (no initialization needed)
        
        # Track position of maximum score
        max_score = 0
        max_pos = (0, 0)
        
        # Fill matrix
        for i in range(1, rows):
            for j in range(1, cols):
                diag = score[i-1][j-1] + self.sub_matrix[self.seq1[j-1], self.seq2[i-1]]
                up = score[i-1][j] + self.gap
                left = score[i][j-1] + self.gap
                
                # Take maximum, but floor at 0
                score[i][j] = max(diag, up, left, 0)
                
                # Record direction (only if not 0)
                if score[i][j] == 0:
                    trace[i][j] = -1  # stop here
                elif score[i][j] == diag:
                    trace[i][j] = 0
                elif score[i][j] == up:
                    trace[i][j] = 1
                else:
                    trace[i][j] = 2
                
                # Update max position
                if score[i][j] > max_score:
                    max_score = score[i][j]
                    max_pos = (i, j)
        
        self.sw_score = score
        self.sw_trace = trace
        self.sw_max_pos = max_pos
        return max_score

Add local alignment backtracking

    def sw_traceback(self):
        aligned1 = ""
        aligned2 = ""
        
        # Start from maximum position
        i, j = self.sw_max_pos
        
        # Stop when we hit 0
        while i > 0 and j > 0 and self.sw_trace[i][j] != -1:
            if self.sw_trace[i][j] == 0:
                aligned1 = self.seq1[j-1] + aligned1
                aligned2 = self.seq2[i-1] + aligned2
                i -= 1
                j -= 1
            elif self.sw_trace[i][j] == 1:
                aligned1 = "-" + aligned1
                aligned2 = self.seq2[i-1] + aligned2
                i -= 1
            else:
                aligned1 = self.seq1[j-1] + aligned1
                aligned2 = "-" + aligned2
                j -= 1
        
        return aligned1, aligned2

Usage

seq_pair = SeqPair("TGA", "GA", tt_matrix, gap_penalty=-2)
score = seq_pair.smith_waterman()
print(f"Score: {score}")  # Score: 4

aln1, aln2 = seq_pair.sw_traceback()
print(aln1)  # GA
print(aln2)  # GA

9. Global vs Local: When to Use Which

Needleman-Wunsch (Global):

• Use when comparing sequences of similar length
• Use when you expect the sequences to be related across their entire length
• Example: comparing two versions of the same gene

Smith-Waterman (Local):

• Use when looking for conserved regions within longer sequences
• Use when sequences have very different lengths
• Example: finding a motif within a larger protein

10. Implementation Classes

AlnSeq Class

Holds two sequences and computes:

• Hamming distance
• Hamming similarity
• Levenshtein similarity (for pre-aligned sequences)
• Custom str for visual output:

  ACTG
  |X|^
  AGTG-
  

  Where: | = match, X = substitution, ^ = insertion, v = deletion

SubstitutionMatrix Class

• Stores the scoring matrix values
• Implements getitem for easy access: matrix['A', 'T'] returns the score

SeqPair Class

• Computes alignment matrices (Needleman-Wunsch and Smith-Waterman)
• Reconstructs alignments through backtracking

Implementation: Extend AlnSeq with Levenshtein and str

Levenshtein similarity for aligned sequences

This assumes sequences are already aligned (may contain gaps).

class AlnSeq:
    # ... previous methods ...
    
    def levenshtein_similarity(self, sub_matrix, gap_penalty=-2):
        # For aligned sequences (with gaps already inserted)
        if len(self.seq1) != len(self.seq2):
            raise ValueError("Aligned sequences must have equal length")
        
        score = 0
        for i in range(len(self.seq1)):
            c1 = self.seq1[i]
            c2 = self.seq2[i]
            
            if c1 == '-' or c2 == '-':
                # Gap penalty
                score += gap_penalty
            else:
                # Use substitution matrix
                score += sub_matrix[c1, c2]
        
        return score

Custom str method for visualization

    def __str__(self):
        if len(self.seq1) != len(self.seq2):
            return f"{self.seq1}\n{self.seq2}"
        
        # Build match string
        match_str = ""
        for i in range(len(self.seq1)):
            c1 = self.seq1[i]
            c2 = self.seq2[i]
            
            if c1 == c2:
                match_str += "|"      # identity
            elif c1 == '-':
                match_str += "^"      # insertion (gap in seq1)
            elif c2 == '-':
                match_str += "v"      # deletion (gap in seq2)
            else:
                match_str += "X"      # substitution
        
        return f"{self.seq1}\n{match_str}\n{self.seq2}"

Usage

aln = AlnSeq("TCA", "T-A")
print(aln)
# Output:
# TCA
# |v|
# T-A

print(aln.levenshtein_similarity(tt_matrix, gap_penalty=-2))  # 2 + (-2) + 2 = 2

Complete AlnSeq Class

class AlnSeq:
    def __init__(self, seq1, seq2):
        self.seq1 = seq1
        self.seq2 = seq2
    
    def hamming_distance(self):
        if len(self.seq1) != len(self.seq2):
            raise ValueError("Sequences must have equal length")
        distance = 0
        for i in range(len(self.seq1)):
            if self.seq1[i] != self.seq2[i]:
                distance += 1
        return distance
    
    def hamming_similarity(self):
        if len(self.seq1) != len(self.seq2):
            raise ValueError("Sequences must have equal length")
        score = 0
        for i in range(len(self.seq1)):
            if self.seq1[i] == self.seq2[i]:
                score += 1
            else:
                score -= 1
        return score
    
    def hamming_similarity_matrix(self, sub_matrix):
        if len(self.seq1) != len(self.seq2):
            raise ValueError("Sequences must have equal length")
        score = 0
        for i in range(len(self.seq1)):
            score += sub_matrix[self.seq1[i], self.seq2[i]]
        return score
    
    def levenshtein_similarity(self, sub_matrix, gap_penalty=-2):
        if len(self.seq1) != len(self.seq2):
            raise ValueError("Aligned sequences must have equal length")
        score = 0
        for i in range(len(self.seq1)):
            c1 = self.seq1[i]
            c2 = self.seq2[i]
            if c1 == '-' or c2 == '-':
                score += gap_penalty
            else:
                score += sub_matrix[c1, c2]
        return score
    
    def __str__(self):
        if len(self.seq1) != len(self.seq2):
            return f"{self.seq1}\n{self.seq2}"
        match_str = ""
        for i in range(len(self.seq1)):
            c1 = self.seq1[i]
            c2 = self.seq2[i]
            if c1 == c2:
                match_str += "|"
            elif c1 == '-':
                match_str += "^"
            elif c2 == '-':
                match_str += "v"
            else:
                match_str += "X"
        return f"{self.seq1}\n{match_str}\n{self.seq2}"

Complete SubstitutionMatrix Class

class SubstitutionMatrix:
    def __init__(self, matrix_dict):
        self.matrix = matrix_dict
    
    def __getitem__(self, key):
        res1, res2 = key
        return self.matrix[res1][res2]

# Transition/Transversion Matrix
tt_matrix = SubstitutionMatrix({
    'A': {'A': 2, 'T': -1, 'C': -1, 'G': 0},
    'T': {'A': -1, 'T': 2, 'C': 0, 'G': -1},
    'C': {'A': -1, 'T': 0, 'C': 2, 'G': -1},
    'G': {'A': 0, 'T': -1, 'C': -1, 'G': 2}
})

Complete SeqPair Class

class SeqPair:
    def __init__(self, seq1, seq2, sub_matrix, gap_penalty=-2):
        self.seq1 = seq1
        self.seq2 = seq2
        self.sub_matrix = sub_matrix
        self.gap = gap_penalty
    
    def _init_matrix(self, rows, cols):
        return [[0] * cols for _ in range(rows)]
    
    # --- Needleman-Wunsch (Global) ---
    def needleman_wunsch(self):
        rows = len(self.seq2) + 1
        cols = len(self.seq1) + 1
        
        score = self._init_matrix(rows, cols)
        trace = self._init_matrix(rows, cols)
        
        for j in range(1, cols):
            score[0][j] = score[0][j-1] + self.gap
            trace[0][j] = 2
        
        for i in range(1, rows):
            score[i][0] = score[i-1][0] + self.gap
            trace[i][0] = 1
        
        for i in range(1, rows):
            for j in range(1, cols):
                diag = score[i-1][j-1] + self.sub_matrix[self.seq1[j-1], self.seq2[i-1]]
                up = score[i-1][j] + self.gap
                left = score[i][j-1] + self.gap
                
                score[i][j] = max(diag, up, left)
                
                if score[i][j] == diag:
                    trace[i][j] = 0
                elif score[i][j] == up:
                    trace[i][j] = 1
                else:
                    trace[i][j] = 2
        
        self.nw_score = score
        self.nw_trace = trace
        return score[rows-1][cols-1]
    
    def nw_traceback(self):
        aligned1 = ""
        aligned2 = ""
        i = len(self.seq2)
        j = len(self.seq1)
        
        while i > 0 or j > 0:
            if i > 0 and j > 0 and self.nw_trace[i][j] == 0:
                aligned1 = self.seq1[j-1] + aligned1
                aligned2 = self.seq2[i-1] + aligned2
                i -= 1
                j -= 1
            elif i > 0 and self.nw_trace[i][j] == 1:
                aligned1 = "-" + aligned1
                aligned2 = self.seq2[i-1] + aligned2
                i -= 1
            else:
                aligned1 = self.seq1[j-1] + aligned1
                aligned2 = "-" + aligned2
                j -= 1
        
        return aligned1, aligned2
    
    # --- Smith-Waterman (Local) ---
    def smith_waterman(self):
        rows = len(self.seq2) + 1
        cols = len(self.seq1) + 1
        
        score = self._init_matrix(rows, cols)
        trace = self._init_matrix(rows, cols)
        
        max_score = 0
        max_pos = (0, 0)
        
        for i in range(1, rows):
            for j in range(1, cols):
                diag = score[i-1][j-1] + self.sub_matrix[self.seq1[j-1], self.seq2[i-1]]
                up = score[i-1][j] + self.gap
                left = score[i][j-1] + self.gap
                
                score[i][j] = max(diag, up, left, 0)
                
                if score[i][j] == 0:
                    trace[i][j] = -1
                elif score[i][j] == diag:
                    trace[i][j] = 0
                elif score[i][j] == up:
                    trace[i][j] = 1
                else:
                    trace[i][j] = 2
                
                if score[i][j] > max_score:
                    max_score = score[i][j]
                    max_pos = (i, j)
        
        self.sw_score = score
        self.sw_trace = trace
        self.sw_max_pos = max_pos
        return max_score
    
    def sw_traceback(self):
        aligned1 = ""
        aligned2 = ""
        i, j = self.sw_max_pos
        
        while i > 0 and j > 0 and self.sw_trace[i][j] != -1:
            if self.sw_trace[i][j] == 0:
                aligned1 = self.seq1[j-1] + aligned1
                aligned2 = self.seq2[i-1] + aligned2
                i -= 1
                j -= 1
            elif self.sw_trace[i][j] == 1:
                aligned1 = "-" + aligned1
                aligned2 = self.seq2[i-1] + aligned2
                i -= 1
            else:
                aligned1 = self.seq1[j-1] + aligned1
                aligned2 = "-" + aligned2
                j -= 1
        
        return aligned1, aligned2

Resources

The exercises and examples in this material are inspired by several open educational resources released under Creative Commons licenses. Instead of referencing each one separately throughout the notes, here is a list of the main books and sources I used:

• [A Practical Introduction to Python Programming- © 2015 Brian Heinold] (CC BY-NC-SA 3.0)

All credit goes to the original authors for their openly licensed educational content.

Note

This course courageously comes without dedicated notes, as most concepts overlap with material from other courses, so cross-referencing should do the job.

For the upcoming project, if anyone needs help with logistics, tool setup, or general organization (fully within the code of conduct), I’m happy to help.

Best of luck.

Biomedical Databases (Protected)

Hey! Welcome to my notes for the Biomedical Databases course where biology meets data engineering.

Course Overview

Supplementary Learning Resource

If you want to dive deeper into database fundamentals (and I mean really deep), check out:

CMU 15-445/645: Intro to Database Systems (Fall 2024)

About the CMU Course

This is one of the best database courses available online, taught by Andy Pavlo at Carnegie Mellon University. It's more advanced and assumes some C++ knowledge, but the explanations are incredibly clear.

The CMU course covers database internals, query optimization, storage systems, and transaction management at a much deeper level. It's perfect if you're curious about how databases actually work under the hood.

Everything

What is database? A database is a large structured set of persistent data, usually in computer-readable form.

A DBMS is a software package that enables users:

• to access the data
• to manipulate (create, edit, link, update) files as needed
• to preserve the integrity of the data
• to deal with security issues (who should have access)

PubMed/MeSH

it comprises more than 39 million citations for biomedical and related journal from MEDLINE, life science journals, and online books

MeSH database (Medical Subject Headings) – controlled vocabulary thesaurus

The query is very easy, just be carefull for what OR, AND and '()'. Read the query correctly to know what is the correct query and correct Mesh.

PDB

Definition: What is PDB? PDB (Protein Data Bank) is the main global database that stores 3D structures of proteins, DNA, RNA, and their complexes.

How experimental structure data is obtained? (3 methods)

1. X-ray Crystallography(88%): uses crystals + X-ray diffraction to map atomic positions.
2. NMR Spectroscopy(10%): uses magnetic fields to determine structures in solution.
3. Cryo-Electron Microscopy (Cryo-EM)(1%)

What is Resolution (Å)? Resolution (in Ångström) measures the level of detail; smaller value = sharper, more accurate structure.

SIFTS (Structure Integration with Function, Taxonomy and Sequence) provides residue-level mapping between:

1. PDB entries ↔ UniProt sequences
2. Connections to: GO, InterPro, Pfam, CATH, SCOP, PubMed, Ensembl

This is how you can search PDB by Pfam domain or UniProt ID.

Method Comparison Summary

AlphaFold

What is AlphaFold? A deep learning system that predicts protein structure from Amino acid sequence.

At CASP14 (2020), AlphaFold2 scored ~92 GDT (Global Distance Test).

AlphaFold essentially solved the protein folding problem for single domains.

pLDDT (predicted Local Distance Difference Test): Stored in the B-factor column of AlphaFold PDB files.

What pLDDT measures: Confidence in local structure (not global fold).

1. Identify structured domains vs disordered regions
2. Decide which parts to trust

PAE (Predicted Aligned Error) Dark blocks on diagonal: Confident domains Off-diagonal dark blocks: Confident domain-domain interactions Light regions: Uncertain relative positions (domains may be connected but orientation unknown)

Use PAE for: Determining if domain arrangements are reliable.

PDB file format: Legacy and mmCIF Format (current standard)

The B-factor Column

The B-factor means different things depending on the method:

When validate you measure:

1. Resolution (for X-ray/Cryo-EM)
2. R-factors (for X-ray)
3. Geometry (for all)

R-factor (X-ray only): Measures how well the model fits the experimental data. <0.20 -> Good fit

Types of R-factors:

1. R-work: Calculated on data used for refinement
2. R-free: Calculated on test set NOT used for refinement (more honest)

R-free is more reliable. If R-work is much lower than R-free, the model may be overfitted.

Data Validation:

1. Resolution
2. Geometery
3. R-Factor

Key Search Fields

Comparing Experimental vs AlphaFold Structures

When AlphaFold structures are available:

Key insight: Low-confidence AlphaFold regions often correspond to regions missing in experimental structures — both are telling you the same thing (disorder/flexibility).

For the Oral Exam

Be prepared to explain:

1. Why crystallography needs crystals — signal amplification from ordered molecular packing

2. The phase problem — you measure amplitudes but lose phases; must determine indirectly

3. What resolution means — ability to distinguish fine details; limited by crystal order

4. Why Cryo-EM grew so fast — no crystals needed, good for large complexes, computational advances

5. NMR gives ensembles, not single structures — restraints satisfied by multiple conformations

6. What pLDDT means — local prediction confidence, stored in B-factor column

7. Difference between pLDDT and PAE — pLDDT is local confidence, PAE is relative domain positioning

8. How to assess structure quality — resolution, R-factors, validation metrics

9. B-factor means different things — mobility (X-ray), fluctuation (NMR), confidence (AlphaFold)

10. How to construct complex PDB queries — combining method, resolution, organism, domain annotations

UniProt

What it gives you:

1. Protein sequences and functions
2. Domains, families, PTMs
3. Disease associations and variants
4. Subcellular localization
5. Cross-references to 180+ external databases
6. Proteomes for complete organisms
7. BLAST, Align, ID mapping tools

                    UniProt
                       │
       ┌───────────────┼───────────────┐
       │               │               │
   UniProtKB        UniRef         UniParc
   (Knowledge)    (Clusters)      (Archive)
       │
   ┌───┴───┐
   │       │
Swiss-Prot TrEMBL
(Reviewed) (Unreviewed)

UniProt classifies how confident we are that a protein actually exists. Query syntax: existence:1 (for protein-level evidence)

It also has ID Mapping: Convert between ID systems

TL;DR

• UniProt = protein database = Swiss-Prot (reviewed, high quality) + TrEMBL (unreviewed, comprehensive)
• Always add reviewed:true when you need reliable annotations
• Query syntax: field:value with AND, OR, NOT
• Use parentheses to group OR conditions properly
• Common fields: organism_id, ec, reviewed, existence, database, proteome, go
• Wildcards: Use * for EC numbers (e.g., ec:3.4.21.*)
• Protein existence: Level 1 = experimental evidence, Level 5 = uncertain

NCBI

What is NCBI? National Center for Biotechnology Information — created in 1988 as part of the National Library of Medicine (NLM) at NIH, Bethesda, Maryland.

What it gives you:

1. GenBank (primary nucleotide sequences)
2. RefSeq (curated reference sequences)
3. Gene database (gene-centric information)
4. PubMed (literature)
5. dbSNP, ClinVar, OMIM (variants & clinical)
6. BLAST (sequence alignment)
7. And ~40 more databases, all cross-linked

TL;DR

• NCBI = US hub for biological databases (GenBank, RefSeq, Gene, PubMed, etc.)
• GenBank = primary archive (raw submissions) vs RefSeq = curated reference (cleaned up)
• RefSeq prefixes: NM/NP = curated, XM/XP = predicted — prefer N* for reliable analysis
• Boolean operators MUST be UPPERCASE: AND, OR, NOT
• Use quotes around multi-word terms: "homo sapiens"[Organism]
• Gene database = best starting point for gene-centric searches
• Properties = what it IS, Filters = what it's LINKED to

Ensembl

Ensembl is a genome browser and database jointly run by the EBI (European Bioinformatics Institute) and the Wellcome Trust Sanger Institute since 1999. Think of it as Google Maps, but for genomes.

What it gives you:

1. Gene sets (splice variants, proteins, ncRNAs)
2. Comparative genomics (alignments, protein trees, orthologues)
3. Variation data (SNPs, InDels, CNVs)
4. BioMart for bulk data export
5. REST API for programmatic access
6. Everything is open source

BioMart: Bulk Data Queries

Workflow Example: ID Conversion Goal: Convert RefSeq protein IDs to Ensembl Gene IDs

TL;DR

• Ensembl = genome browser + database for genes, transcripts, variants, orthologues
• IDs: ENSG (gene), ENST (transcript), ENSP (protein) — learn to recognize them
• MANE Select = highest quality transcript annotation (use these when possible)
• BioMart = bulk query tool: Dataset → Filters → Attributes → Export

Avoid these mistakes:

1. Don't paste RefSeq/UniProt IDs in "Gene stable ID" field — use EXTERNAL filters
2. Use the text input field, not just checkboxes
3. Orthologue = cross-species, Paralogue = same species
4. Start with the species of your INPUT IDs as your dataset
5. Always include your filter column in output attributes M

Boolean Algebra in Nutshell

There are only two Boolean values:

• True (1, yes, on)
• False (0, no, off)

Basic Operators

AND Operator (∧)

The AND operator returns True only when both inputs are True.

Truth Table:

OR Operator (∨)

The OR operator returns True when at least one input is True.

Truth Table:

NOT Operator (¬)

The NOT operator flips the value - True becomes False, False becomes True.

Truth Table:

Combining Operators

You can combine operators to create complex logical expressions.

Operator Precedence (Order of Operations)

Example: A OR B AND C

• First do: B AND C
• Then do: A OR (result)

Use parentheses to be clear: (A OR B) AND C