Magic Commands IPython

Warning : This is not to be confused with the official Magic functions of Python.

IPython magic commands are a special piece of syntax used in Python programming commonly inside Jupyter / Google Colab notebooks to be able to solve various complex problems faced by developers in their day-to-day lives using the Python programming language. Prefixed with a %, these commands mainly act as an enhancement to the already existing Python programming language syntax and features. These commands can be mainly used to control the behaviour of IPython itself and also gain control of system-type features.

Magic commands can be categorized into 2 main classes:

Line Magics

Line Magic also denoted by % or a single percentage symbol generally works on single-lined inputs. Now, we will be discussing a few of the most commonly used commands under this,
1. Magic
```
 %magic
```
  The above command will provide general documentation about the different commands that can be used with the IPython magic syntax. The %lsmagic command can also be used to get a more comprehensive overview of the other commands present and it will print out all of the available magic commands.
2. TimeIt
```
 %timeit range(3000)
```
  This command can be used to find the average amount of time taken for a single piece of code block to execute. A more comprehensive example can be -
```
 %timeit
 for i in range(100):
     i**2
```
  The above code outputs 20.2 µs ± 731 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each). The number of loops and iterations can also be specified using -n and -r. For example: %timeit -n 100 -r 10 sum(range(100))
3. Alias
```
 %alias alias_name cmd
```
  This command can be used to set up aliases with the syntax %alias <alias-name> <command>. This proves to be particularly helpful in customizing and configuring complex commands. For example, %alias ll ls -l can be used to set up an alias for a slightly complex command like ls -l as ll which can be used easily and can be used to simplify commands that are much more complex as well.
4. AutoAwait
```
 %autoawait on
```
  This command can be useful to automatically await the result of an asynchronous command in Jupyter notebooks.
5. BookMark
```
 %bookmark
```
  This command will let you interact with the bookmarking system in IPython environments. It can be particularly useful to save your directories if you find yourself switching between a lot of them. For example, %pwd followed by %bookmark book will save the current working directory to the bookmark which can be viewed with the command %bookmark -l
6. More
```
 %more <file-path>
```
  The more magic command is used to peek into files without having to call inbuilt commands. For example, a file like mydata.csv could be viewed directly by doing %more mydata.csv
7. Run
```
 %run <file-path>
```
  This more magic command can be used to execute Python code files present elsewhere inside your notebook. For example, let's say you have a Jupyter notebook open with a hello_world.py as well on doing %run hello_world.py the code written inside the Python file will get executed with its output displayed in the notebook.
8. History
```
 %history
```
  The history magic command can be used to display the comprehensive history of the commands executed within the current Jupyter notebook session.
9. Reset
```
 %reset
```
  The reset magic command is used to reset all of the variables/values initialized and used within the current working session inside the notebook.
10. Debug
```
%debug
```
  The debug magic command can be used to activate the interactive Python debugger within the notebook which will help you debug errors/tracebacks.
11. Who
```
%who
```
  The who magic command (%whos is more comprehensive) to get more details on the variables declared and being used within the current notebook session.
Cell Magics

Cell Magics is a class of the magics command denoted by a %% or a double percentage symbol can be used to run commands that span across multiple lines. Some of the commonly used commands under this are,

WriteFile
```
 %%writefile <path>
```
The writefile magic command can be used to write the required output stored inside a variable etc into a file. For example, %%writefile example.txt followed by a string like "Hello World!" will write the string to the file.
Capture
```
 %%capture <variable-name> <code>
```
The capture code magic can be very useful to store the stderr/traceback of a block of code inside a variable you have defined. For example,
```
 %%capture capt
 from __future__ import print_function
 import sys
 print('Hello stdout')
 print('and stderr', file=sys.stderr)
```
```
 capt.stdout, capt.stderr
```
The above code will store the values 'Hello stdout' and 'and stderr' or else the capt.show() command can also be used to display all of the outputs in the form of a tuple ('Hello stdout', 'and stderr') .
Language-Based Interpreter CodeMagic
```
 %%<language-name>
```
This is a unique feature of cell-based code magics where %%<language-name> will convert your current cell into an interpreter for a programming language of your desire. For example, %%bash followed by echo "hello world $BASH" will output hello world /bin/bash.

P.S. Run the %lsmagic% command to get the list of all programming languages that can be used with this command. And, these scripts can be converted to background scripts by using the --bg flag after the language name.
Prun
```
 %%prun
```
The Prun magic command is a game changer and is highly useful for data scientists/developers who mainly want to gain analytical insights or analyse the performance of their code. For example,
```
 %%prun
 def my_function():
     for i in range(1000000):
         pass

 my_function()
```
On running the above code, you will get an output displaying the time taken / the memory utilized to run the above code.
Debug
```
 %%debug
```
The debug command can be utilized to debug your code interactively. For example,
```
 %%debug
 def divide(a, b):
     result = a / b
     return result

 divide(10, 0)
```
In the above code, if a division with 0 is performed, the debugger is activated which will let you step through your code and inspect the variables to analyse the problem being caused.

Thus, these are some of the commands used quite frequently (with a few lesser-known ones) in IPython environments to improve your development workflow especially when working with Google Colab / Jupyter Notebook environments.

Now, before you go to read your next article since you've read this article so far, I'd like to enlighten you with a special feature of magic commands.

This is the ability to set up custom magic commands to simplify tasks that would normally take you a long time to set up. For example, let's consider the below complex code :

import secrets
from IPython.core.magic import register_line_magic

@register_line_magic
def reset_passwords(line):
    new_password = line or secrets.token_urlsafe(10)
    all_users = User.objects.all()

    if not all_users:
        print("No users found.")
        return

    num_users = 0
    for user in all_users:
        try:
            user.set_password(new_password)
            user.save()
            num_users += 1
        except Exception as e:
            print(f"Failed to reset password for user {user.username}: {str(e)}")

    print(f'{num_users} passwords were changed to "{new_password}"')

The above code is used to set up application-specific logic to reset passwords. However, imagine having to rewrite this function multiple times. Instead of that, the above function can be stored in the form of a script in your device which you can then leverage to call around multiple Jupyter notebooks and get it to execute.

To register custom magic commands with the IPython instance, you need to use the @register_line_magic or @register_cell_magic decorators provided by IPython. These decorators allow you to define your very own magic functions and register them with IPython. Once registered, you can use your custom magic commands in the same way as the built-in magic commands.

These scripts are to be saved on your local machine's .ipython directory with a name. After doing this, similar to a function call, the script can be called with the script name on any notebook of your choice and executed.

In conclusion, IPython magic commands offer a powerful toolset for enhancing productivity, especially for Machine Learning engineers and data scientists. These commands provide a means to control the behaviour of IPython and gain control of system-type features, thereby simplifying complex tasks and improving the overall development workflow. While these commands may not be as widely recognized as other aspects of Python, their utility is undeniable. I strongly encourage you to explore these commands and more (yes there are more which aren't covered).

Thank you for your time in reading this article! Good day!

IPython Magic Commands

Warning : This is not to be confused with the official Magic functions of Python.

Did you find this article valuable?