5. Advance Python Misc

1. Progress Bar

In Python, you can display a progress bar in the command line using various libraries or by creating a simple custom implementation. Here are a few methods to achieve this:

1.1 Method 1: Using tqdm Library

The tqdm library is a popular choice for creating progress bars in Python. It’s easy to use and highly customizable.

Installation

First, you need to install tqdm if you haven’t already:

pip install tqdm

Example

Here’s how to use tqdm to display a progress bar:

from tqdm import tqdm
import time

# Simulate a process with a loop
for i in tqdm(range(100), desc="Processing"):
    time.sleep(0.1)  # Simulating work by sleeping for 0.1 seconds

Another Example

To use tqdm with a custom generator in Python, you can wrap the generator in tqdm while iterating over it. Here’s an example of how to create a custom generator and integrate it with tqdm:

from tqdm import tqdm
import time

# Custom generator function
def custom_generator(limit):
    for i in range(limit):
        yield i

# Wrapping the generator with tqdm
for number in tqdm(custom_generator(100), total=100):
    # Simulating some work with a sleep
    time.sleep(0.1)  # Replace with your actual processing

Explanation:

• Custom Generator: The custom_generator yields numbers up to the limit you provide.
• tqdm with Generator: When using tqdm with a generator, you need to specify the total argument, as tqdm won’t know the number of items to process from the generator itself.
• Iterating and Progress Display: tqdm displays the progress as you iterate over the generator. The time.sleep(0.1) simulates work.

This approach is flexible and works well with any custom generator.

1.2 Method 2: Custom Progress Bar

If you prefer not to use an external library, you can create a simple progress bar using basic Python code:

Example

import sys
import time

def print_progress_bar(iteration, total, prefix='', length=40, fill='█'):
    percent = (iteration / total) * 100
    filled_length = int(length * iteration // total)
    bar = fill * filled_length + '-' * (length - filled_length)
    sys.stdout.write(f'\r{prefix} |{bar}| {percent:.2f}% Complete')
    sys.stdout.flush()

# Simulate a process with a loop
total = 100
for i in range(total):
    time.sleep(0.1)  # Simulating work
    print_progress_bar(i + 1, total, prefix='Progress')

print()  # Move to the next line after completion

1.3 Explanation:

1. Using tqdm:

   • Simply wrap any iterable (like a range) with tqdm() to get a progress bar automatically.

2. Custom Progress Bar:

   • print_progress_bar function: This function constructs and displays a progress bar based on the current iteration and total.
   • sys.stdout.write: This is used to print to the same line in the console.
   • time.sleep(0.1): Simulates a task that takes time.

Tip

Using tqdm is the simplest way to implement a progress bar in Python, but if you prefer a custom solution, the second method gives you flexibility. Both methods work well for providing visual feedback during long-running tasks in command-line applications.

2. Logger

The logging module in Python provides a flexible framework for logging messages from your programs.

2.1 Why use logging over print?

While print() is simple for debugging, logging is much more powerful because:

• It supports different log levels (e.g., INFO, DEBUG, ERROR).
• You can configure where the logs are stored (e.g., console, file, network).
• You can customize the log format.
• You can control logging for different modules in a large application.
• You can turn off logging in production without changing the code.

2.2 Basic Logging Setup

Here’s how to set up and use the logging module with minimal configuration:

import logging

# Set up basic configuration
logging.basicConfig(level=logging.DEBUG)

# Log messages at different levels
logging.debug('This is a debug message')
logging.info('This is an info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')

• Output:

  WARNING:root:This is a warning message
  ERROR:root:This is an error message
  CRITICAL:root:This is a critical message

Notice that only WARNING and higher levels are printed by default unless the level is set to DEBUG (as in the example).

2.3 Logging Levels

Python’s logging module has six built-in levels:

Level	Value	Description
DEBUG	10	Detailed information, typically for diagnosing problems.
INFO	20	Confirmation that things are working as expected.
WARNING	30	Something unexpected, but the program continues working.
ERROR	40	More serious problem; the program may not be able to continue.
CRITICAL	50	A very serious error; program execution may stop.
NOTSET	0	When no level is set.

You set the logging level using logging.basicConfig(level=logging.<LEVEL>). For example, setting it to logging.DEBUG allows all messages (DEBUG, INFO, etc.) to be displayed.

---

2.4 Loggers, Handlers, and Formatters

• Loggers: The main entry point of the logging system, responsible for capturing log messages.
• Handlers: These send the log messages to a particular destination (console, file, etc.).
• Formatters: Define the layout of the log messages (e.g., timestamps, log levels).

Here’s how to use them:

import logging

# Create a custom logger
logger = logging.getLogger('my_logger')
logger.setLevel(logging.DEBUG)

# Create handlers
console_handler = logging.StreamHandler()  # Log to console
file_handler = logging.FileHandler('app.log')  # Log to file

# Set levels for handlers
console_handler.setLevel(logging.WARNING)  # Console logs warnings and above
file_handler.setLevel(logging.DEBUG)  # File logs all levels

# Create formatters
console_format = logging.Formatter('%(name)s - %(levelname)s - %(message)s')
file_format = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# Add formatters to handlers
console_handler.setFormatter(console_format)
file_handler.setFormatter(file_format)

# Add handlers to logger
logger.addHandler(console_handler)
logger.addHandler(file_handler)

# Log messages
logger.debug('This is a debug message')
logger.info('This is an info message')
logger.warning('This is a warning message')
logger.error('This is an error message')
logger.critical('This is a critical message')

• Output on Console (only WARNING and above):

  my_logger - WARNING - This is a warning message
  my_logger - ERROR - This is an error message
  my_logger - CRITICAL - This is a critical message

• Content of app.log (all levels):

  2024-10-07 12:00:00,000 - my_logger - DEBUG - This is a debug message
  2024-10-07 12:00:00,001 - my_logger - INFO - This is an info message
  2024-10-07 12:00:00,002 - my_logger - WARNING - This is a warning message
  2024-10-07 12:00:00,003 - my_logger - ERROR - This is an error message
  2024-10-07 12:00:00,004 - my_logger - CRITICAL - This is a critical message

2.5 Writing Logs to a File

If you want to log messages to a file, use the FileHandler:

import logging

# Basic configuration to log to file
logging.basicConfig(filename='app.log', level=logging.DEBUG,
                    format='%(asctime)s - %(levelname)s - %(message)s')

logging.debug('This will go to the file')
logging.error('This is an error in the log file')

This logs all messages (since DEBUG level is set) to app.log.

2.6 Configuring Multiple Loggers

You may want to set up multiple loggers, for instance, if you are logging different parts of a large system.

import logging

# Logger for module 1
logger1 = logging.getLogger('module1')
logger1.setLevel(logging.DEBUG)
handler1 = logging.FileHandler('module1.log')
logger1.addHandler(handler1)

# Logger for module 2
logger2 = logging.getLogger('module2')
logger2.setLevel(logging.ERROR)
handler2 = logging.FileHandler('module2.log')
logger2.addHandler(handler2)

# Log messages
logger1.debug('Debug message from module1')
logger2.error('Error message from module2')

• module1.log will contain debug messages from module1.
• module2.log will only contain error messages from module2.

2.7 Advanced Features

• Rotating log files: You can rotate logs when they reach a certain size using RotatingFileHandler to keep the logs from growing indefinitely.

  from logging.handlers import RotatingFileHandler
  
  handler = RotatingFileHandler('app.log', maxBytes=2000, backupCount=5)

• Logging to external systems: You can log messages to other systems, like remote servers, databases, or network log aggregators, by using the appropriate handlers.

Tip

• Basic setup: logging.basicConfig() with different levels and formats.
• Loggers, Handlers, Formatters: Use them to customize the log behavior and format.
• Multiple loggers: For more control over different parts of an application.
• Advanced logging: Rotating files, logging to remote systems, etc.

Using logging in Python allows you to track and manage the flow of your program and make debugging, monitoring, and auditing much easier.

3. Python with Excel Files

In Python, when reading and writing Excel files using libraries like pandas or openpyxl, formatting is usually lost because these libraries primarily focus on the data rather than the formatting.

To preserve the formatting, you can use openpyxl to load the existing file and make modifications without altering the format. Here’s how to do it:

1. Use pandas to manipulate the data (add a new column).
2. Use openpyxl to load the original file and save the modified data, while keeping the formatting intact.

Here’s an example:

import pandas as pd
from openpyxl import load_workbook

# Step 1: Load the Excel file using pandas
df = pd.read_excel('your_file.xlsx')

# Step 2: Calculate the new column
df['new_column'] = df['existing_column'] * 2  # Example calculation

# Step 3: Write the updated data back without formatting
with pd.ExcelWriter('your_file.xlsx', engine='openpyxl', mode='a', if_sheet_exists='overlay') as writer:
    # Write the modified data back to the original file
    df.to_excel(writer, sheet_name='Sheet1', index=False)

This approach uses the openpyxl engine with pandas and the mode='a' (append mode) option to ensure the original formatting remains intact.

4. Generators

In Python, you can explicitly stop a generator that uses yield by raising an exception, such as StopIteration. Here’s an example:

def my_generator():
    yield 1
    yield 2
    raise StopIteration  # Stops the generator
    yield 3

gen = my_generator()

for item in gen:
    print(item)

In this case, the generator will stop after yielding 2 due to the StopIteration being raised.

Alternatively, you can return from the generator to stop it:

def my_generator():
    yield 1
    yield 2
    return  # Explicitly stop the generator
    yield 3  # This line will never execute

gen = my_generator()

for item in gen:
    print(item)

This will also stop after yielding 2 because the return ends the generator.