🧠 Python-within-Python Script Execution Methods

As you know, I like to break my input into several smaller files. While packaging code into functions is great for reusability, it doesn’t always suit my workflow. Often, I just want to repeat a specific sequence of commands multiple times throughout a script without the overhead of managing input and output lists.

To keep things simple, I save these code chunks as separate scripts and then "source" them into my main script whenever needed. I started using this approach years ago when working on the OpenSees Examples Manual, and it’s since become a standard part of how I, and others in the OpenSees community, organize my models and analyses.

Here is what I got from ChatGPT, which has a way of filling in my knowledge gaps. There are two parts to this post, as I asked a follow-up question about which cases can run as separate processes..

Part 1 -- Description

There are several ways to execute Python code from another file. Each method behaves differently in terms of scope, output, portability, and use case.

✅ 1. %run (Jupyter/IPython Magic)

Usage:

%run my_script.py

Features:

• Runs another Python script in the current namespace

• Works only in Jupyter or IPython

• Variables and functions defined in the script are accessible afterward

Pros:

• Simple and interactive

• Great for chaining notebooks or scripts

• Variables/functions persist in your environment

Cons:

• Only works in IPython/Jupyter

• Can pollute your namespace

  
  

✅ 2. os.system() (Standard Python)

Usage:

import os
os.system('python my_script.py')

Features:

• Runs the script in a separate shell process

• No variable sharing between scripts

• Only captures output if redirected

Pros:

• Simple and universal (works anywhere)

• Can run any shell command, not just Python

Cons:

• No interaction with the script's variables

• Hard to capture or manage output and errors

  
  

✅ 3. exec(open().read()) (Inline Execution)

Usage:

exec(open('my_script.py').read())

Features:

• Reads and executes the entire script as a string

• Runs in the current namespace

Pros:

• Lightweight and fast

• Works in any Python environment

• Variables/functions from the script are available afterward

Cons:

• Less safe (harder to debug or trace errors)

• No control over execution scope or error handling

• Cannot pass command-line arguments

  
  

✅ 4. subprocess.run() (Modern, Flexible)

Usage:

import subprocess
subprocess.run(['python', 'my_script.py'])

Features:

• Runs script as a separate subprocess (like os.system())

• Allows better control over input/output, errors, environment

Pros:

• More powerful and secure than os.system()

• You can capture output, set timeouts, handle errors

• Cross-platform and production-friendly

Cons:

• Slightly more verbose than os.system()

• Still doesn't share variables with the main script

  
  

✅ 5. import (Modular Approach)

Usage:

import my_script
my_script.some_function()

Features:

• Imports another script as a module

• Only runs code under if name == "__main__": when called directly

• Lets you reuse functions, classes, and variables cleanly

Pros:

• Clean, Pythonic, and modular

• Allows selective reuse of code

• Best practice for reusable logic

Cons:

• Doesn't automatically run script body unless wrapped in a function or guarded by main

• Requires the file to be in the Python path

🧾 Summary Table

Method	Runs in Same Namespace?	Works Outside Jupyter?	Captures Output?	Can Access Variables?	Use Case
%run	✅ Yes	❌ No	✅ Yes (inline)	✅ Yes	Jupyter notebooks & interactive use
os.system()	❌ No	✅ Yes	❌ Not by default	❌ No	Shell-style execution
exec(open().read())	✅ Yes	✅ Yes	✅ Yes	✅ Yes	Simple script inclusion
subprocess.run()	❌ No	✅ Yes	✅ Yes (if set)	❌ No	External process control
import	✅ Yes (functions)	✅ Yes	✅ Yes	✅ Yes (modular)	Clean code reuse

🔍 Recommendation

If you want to...	Use this method
Reuse variables/functions interactively	%run or exec()
Run a black-box script externally	os.system() or subprocess.run()
Reuse code cleanly and modularly	import
Automate batch execution or CLI tools	subprocess.run()

Part 2 -- Can these be run as a separate process?

Yes! Some of these methods can run code in a separate process, while others run it in the same process as your current script or notebook.

Here’s a detailed breakdown of whether each method runs in a separate process, and what that means for memory, variable sharing, and isolation.

🧠 What Does “Separate Process” Mean?

Running code in a separate process means:

• It's isolated from your main script (no variable sharing)

• It has its own memory space

• It can crash or run forever without affecting your main script (unless you wait for it)

• Useful for parallelism, automation, and subprocess management

  
  

✅ Process Isolation Summary

Method	Runs in Separate Process?	Shares Variables?	Can Run in Parallel?	Notes
%run	❌ No (same process)	✅ Yes	❌ No	Jupyter-only
exec(open().read())	❌ No (same process)	✅ Yes	❌ No	Lightweight inline execution
import	❌ No (same process)	✅ Yes (functions)	❌ No	Modular, reusable
os.system()	✅ Yes	❌ No	✅ Yes (if managed)	Basic external command runner
subprocess.run()	✅ Yes	❌ No	✅ Yes (more control)	Preferred for external execution
multiprocessing.Process	✅ Yes	❌ No (uses queue/pipes)	✅ Yes	Best for custom Python parallelism

✅ Methods That Run in a Separate Process

1. os.system()

Runs any command in a new shell process.

import os 
os.system('python other_script.py')

• Blocking (waits until it finishes)

• Doesn’t give you much control over output or errors

  
  

2. subprocess.run()

Runs commands in a new subprocess with control over I/O.

import subprocess 
subprocess.run(['python', 'other_script.py'])

You can also do:

result = subprocess.run(['python', 'script.py'], capture_output=True, text=True)
print(result.stdout)

✅ Recommended for serious scripting, automation, and CLI tools.

3. multiprocessing.Process

Runs a Python function in a new process (fully parallel).

from multiprocessing import Process
def task(): 
	print("Running in a separate process") 
p = Process(target=task) 
p.start() 
p.join()

✅ Use this for true parallel execution in Python, not just launching scripts.

❌ Methods That Run in the Same Process

• %run

• exec(open().read())

• import (unless your imported script manually launches processes)

These are not isolated — all variables/functions are shared with your current script.

🧠 When to Use Separate Processes

Use a separate process when:

• You want parallel execution

• You want to avoid namespace pollution

• You need to crash-proof a subprocess

• You're automating batch runs or simulations

  
  

✅ TL;DR – Can They Run as Separate Processes?

Method	Separate Process?	Use Case
%run	❌ No	Jupyter chaining, debugging
exec(open().read())	❌ No	Inline execution, quick tests
import	❌ No	Code reuse, modular design
os.system()	✅ Yes	Simple shell/script automation
subprocess.run()	✅ Yes	Robust process control
multiprocessing	✅ Yes	Parallelism inside Python

.....Hasn't multiprocessing been replaced by concurrent.futures?

... I had to ask.... I use concurrent.futures... but this topic needs its own blog...