GIL Deep Dive - Global Interpreter Lock

Introduction

The Global Interpreter Lock (GIL) is a mutex that protects access to Python objects, preventing multiple threads from executing Python bytecode simultaneously. Understanding GIL helps choose between threading and multiprocessing.

GIL Basics

import threading
import time

# GIL ensures only one thread runs Python at a time
counter = 0

def increment():
    global counter
    for _ in range(10**6):
        counter += 1

threads = [threading.Thread(target=increment) for _ in range(4)]
for t in threads:
    t.start()
for t in threads:
    t.join()

print(counter)  # < 4000000 due to GIL contention

Threading vs Multiprocessing

import threading
import multiprocessing
import time

# Threading - limited by GIL for CPU-bound tasks
def cpu_bound(n):
    result = 0
    for i in range(n):
        result += i ** 2
    return result

# Multiprocessing - bypasses GIL
def cpu_bound_mp(n):
    return cpu_bound(n)

# Threading
start = time.time()
threads = [threading.Thread(target=cpu_bound, args=(10**6,)) for _ in range(4)]
for t in threads:
    t.start()
for t in threads:
    t.join()
print(f"Threading: {time.time() - start:.2f}s")

# Multiprocessing
start = time.time()
with multiprocessing.Pool(4) as pool:
    pool.map(cpu_bound_mp, [10**6] * 4)
print(f"Multiprocessing: {time.time() - start:.2f}s")

GIL and I/O

import threading
import urllib.request

# I/O releases GIL, so threading works well
def fetch_url(url):
    with urllib.request.urlopen(url) as response:
        return response.read()

urls = ["http://example.com"] * 10
threads = [threading.Thread(target=fetch_url, args=(url,)) for url in urls]
for t in threads:
    t.start()
for t in threads:
    t.join()

Why GIL Exists

# GIL provides:
# 1. Simpler memory management (reference counting)
# 2. Faster single-threaded performance
# 3. Easy integration with C extensions

# Thread-safe operations still need locks
counter = 0
lock = threading.Lock()

def safe_increment():
    global counter
    with lock:
        counter += 1

Working Around GIL

import multiprocessing
from concurrent.futures import ProcessPoolExecutor
import ctypes

# Use processes for CPU-bound tasks
def cpu_intensive():
    return sum(i**2 for i in range(10**7))

with ProcessPoolExecutor(max_workers=4) as executor:
    results = list(executor.map(lambda x: cpu_intensive(), range(4)))

# Use C extensions (NumPy, etc.) - they release GIL
import numpy as np
result = np.dot(arr1, arr2)  # Releases GIL

Practice Problems

1. Compare threading vs multiprocessing performance
2. Identify CPU-bound vs I/O-bound tasks
3. Use ProcessPoolExecutor for parallel work
4. Implement thread-safe counter
5. Measure GIL impact on performance