Bash Powertools — Elite Shell Automation

The best LLMs write Bash like senior DevOps engineers: safe, composable, readable,

and idiomatic. This skill is the difference between brittle one-liners and production-grade

automation that runs reliably in IMI's research and data environments.

Script Structure — Every Production Script

#!/usr/bin/env bash
# ============================================================
# Script: process_wave_exports.sh
# Purpose: Process IMI wave export CSVs and archive originals
# Usage: ./process_wave_exports.sh <input_dir> <output_dir>
# Author: IMI Local AI | Date: 2026-03
# ============================================================

set -euo pipefail          # -e: exit on error, -u: unset vars = error, -o pipefail
IFS=$'\n\t'                # safer word splitting

# Constants
readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly LOG_FILE="${SCRIPT_DIR}/logs/process_$(date +%Y%m%d_%H%M%S).log"
readonly MIN_BASH_VERSION=4

# Logging
log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $*" | tee -a "$LOG_FILE"; }
err() { log "ERROR: $*" >&2; }
die() { err "$*"; exit 1; }

# Argument validation
[[ $# -lt 2 ]] && die "Usage: $0 <input_dir> <output_dir>"
INPUT_DIR="$1"
OUTPUT_DIR="$2"
[[ -d "$INPUT_DIR" ]] || die "Input directory not found: $INPUT_DIR"
mkdir -p "$OUTPUT_DIR"

# Main logic here
main() {
    log "Starting processing: $INPUT_DIR → $OUTPUT_DIR"
    # ... work ...
    log "Done."
}

main "$@"

Non-negotiable patterns:

• set -euo pipefail — always. No exceptions.
• readonly for constants
• Validate every argument before using it
• Log with timestamps to a file, not just stdout
• die() function for clean error exit

Text Processing Powertools

grep — the search engine

# Recursive search, show line numbers, context, case-insensitive
grep -rn "brand_consideration" ./data/ --include="*.csv" -C 2

# Count occurrences per file
grep -c "Wave 3" *.csv

# Only show filenames
grep -rl "BRAND_HEALTH_2024" ./studies/

# Invert match (lines NOT containing)
grep -v "^#" config.txt | grep -v "^$"

# Extended regex with groups
grep -E "(Wave [0-9]+|w[0-9]+)" study_*.csv

# Print only the match, not the whole line
grep -oP '(?<=study_id=")[^"]+' config.json

awk — the data transformer

# Sum a column (weighted_n column 4)
awk -F',' 'NR>1 {sum += $4} END {print "Total weighted n:", sum}' results.csv

# Filter rows and transform (pipe-friendly)
awk -F',' 'NR==1 || ($3 >= 100 && $5 > 0.58)' tracking_wave3.csv

# Calculate T2B percentage from raw counts
awk -F',' 'NR>1 {
    t2b = ($4 + $5) / ($3 + $4 + $5 + $6 + $7) * 100
    printf "%s,%.1f\n", $1, t2b
}' response_counts.csv

# Multi-file processing with filename tracking
awk -F',' 'FNR==1 && NR>1 {print "---", FILENAME} NR>1' wave_*.csv

# Reorder columns
awk -F',' 'BEGIN{OFS=","} {print $1,$3,$2,$4}' messy.csv

sed — the stream editor

# In-place replacement (macOS needs -i '')
sed -i 's/respondant/respondent/g' *.csv       # Linux
sed -i '' 's/respondant/respondent/g' *.csv    # macOS

# Delete lines matching pattern
sed '/^#/d; /^$/d' config.txt

# Print only lines 10-20
sed -n '10,20p' large_report.txt

# Multi-expression transformation
sed -e 's/Wave_/Wave /g' -e 's/,,/,NA,/g' data.csv

# Extract between markers
sed -n '/BEGIN_FINDINGS/,/END_FINDINGS/p' report.txt

jq — JSON surgery (essential for API responses, config files)

# Pretty print
cat pulse_response.json | jq '.'

# Extract specific fields
jq '.data[] | {brand: .brand_name, alignment: .alignment_index}' pulse_data.json

# Filter and transform
jq '[.results[] | select(.base_size >= 100) | {passion_point, alignment_index, country}]' pulse.json

# Compute from JSON
jq '[.waves[].consideration_t2b] | add / length' tracking.json

# Output as CSV
jq -r '.data[] | [.brand_name, .passion_point, .alignment_index] | @csv' pulse.json > output.csv

# Merge two JSON files
jq -s '.[0] * .[1]' base.json override.json

File System Operations

Finding files like a surgeon

# Find CSVs modified in the last 7 days
find ./data -name "*.csv" -mtime -7 -type f

# Find large files (> 50MB)
find . -size +50M -type f -exec ls -lh {} \;

# Find and process (safe with -print0/-0 for filenames with spaces)
find ./exports -name "Wave_*.csv" -print0 | xargs -0 -I{} cp {} ./archive/

# Find and rename with date prefix
find ./reports -name "*.pdf" | while IFS= read -r f; do
    dir=$(dirname "$f")
    base=$(basename "$f")
    mv "$f" "${dir}/$(date +%Y%m%d)_${base}"
done

Batch operations

# Rename all files: replace spaces with underscores
for f in ./*.csv; do
    mv "$f" "${f// /_}"
done

# Add wave prefix to all files in a directory
wave="Wave_3_"
for f in exports/*.csv; do
    mv "$f" "exports/${wave}$(basename "$f")"
done

# Archive and compress by date
tar -czf "exports_$(date +%Y%m%d).tar.gz" ./exports/
# Extract
tar -xzf exports_20260301.tar.gz -C ./restored/

Safe deletion with confirmation

# Always preview before deleting
find . -name "*.tmp" -type f
# Then delete after visual confirmation:
find . -name "*.tmp" -type f -delete

# Move to trash instead of hard delete
trash_dir="${HOME}/.trash_$(date +%Y%m%d)"
mkdir -p "$trash_dir"
mv ./old_exports/* "$trash_dir/"

Process Management

# Run in background, capture PID
./long_analysis.py &
PID=$!
echo "Running as PID $PID"

# Wait for completion with timeout
timeout 300 ./analysis.py || die "Analysis timed out after 300s"

# Run multiple jobs in parallel, limit concurrency
process_file() {
    echo "Processing: $1"
    python process.py "$1"
}
export -f process_file
find ./data -name "*.csv" | parallel -j4 process_file {}   # GNU parallel

# Trap cleanup on exit
cleanup() {
    log "Cleaning up temp files..."
    rm -rf "$TMPDIR"
}
trap cleanup EXIT INT TERM

# Check if process is running
if pgrep -f "pulse_updater.py" > /dev/null; then
    echo "Pulse updater already running"
else
    ./pulse_updater.py &
fi

Cron — Scheduled Automation

# View current crontab
crontab -l

# Edit crontab
crontab -e

# Cron syntax: minute hour day month weekday command
# ┌──── minute (0-59)
# │  ┌─ hour (0-23)
# │  │  ┌ day of month (1-31)
# │  │  │ ┌ month (1-12)
# │  │  │ │ ┌ day of week (0=Sun, 6=Sat)
# │  │  │ │ │
  0  7  *  *  1   /home/imi/scripts/weekly_pulse_export.sh >> /var/log/pulse.log 2>&1
  0  9  *  *  1-5 /home/imi/scripts/daily_tracking_check.sh
  30 6  1  *  *   /home/imi/scripts/monthly_report.sh

# Common cron patterns:
# Every 15 minutes:          */15 * * * *
# Every hour:                0 * * * *
# Daily at 7am:              0 7 * * *
# Weekdays at 9am:           0 9 * * 1-5
# First of month at midnight: 0 0 1 * *

# ALWAYS redirect output — silent cron jobs lose errors
0 7 * * * /path/to/script.sh >> /var/log/imi_cron.log 2>&1

Networking and Remote Tools

# curl — API calls, file downloads
curl -s -X POST "https://api.example.com/pulse" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"country": "CA", "wave": "latest"}' | jq '.data'

# Download with retry and timeout
curl -L --retry 3 --retry-delay 5 --max-time 60 \
  -o ./data/pulse_export.csv \
  "https://data.consultimi.com/exports/pulse_latest.csv"

# rsync — better than cp for remote/large transfers
rsync -avz --progress ./exports/ user@server:/data/imi_exports/
# With exclusions
rsync -avz --exclude='*.tmp' --exclude='.DS_Store' ./reports/ backup:/reports/

# SSH tunneling for database access
ssh -L 5432:localhost:5432 user@imi-db-server -N &
# Now connect locally to port 5432 → tunnels to remote postgres

Environment and Secrets Management

# NEVER hardcode credentials — use environment variables
export IMI_DB_PASSWORD="$(cat ~/.secrets/imi_db_pass)"
export PULSE_API_KEY="$(cat ~/.secrets/pulse_api_key)"

# .env file pattern (use with source or python-dotenv)
cat > .env << 'EOF'
IMI_DB_HOST=db.consultimi.com
IMI_DB_PORT=5432
IMI_DB_NAME=research_db
# NEVER commit .env to git
EOF
echo ".env" >> .gitignore

# Load env in script
set -a; source .env; set +a

# Check required env vars before running
required_vars=(IMI_DB_HOST IMI_DB_PORT PULSE_API_KEY)
for var in "${required_vars[@]}"; do
    [[ -z "${!var}" ]] && die "Required environment variable not set: $var"
done

IMI-Specific Automation Patterns

Pulse™ export processing pipeline

#!/usr/bin/env bash
set -euo pipefail

# Process incoming Pulse™ tabular exports
EXPORTS_DIR="./incoming/pulse"
PROCESSED_DIR="./processed/pulse"
ARCHIVE_DIR="./archive/pulse"

mkdir -p "$PROCESSED_DIR" "$ARCHIVE_DIR"

for export_file in "$EXPORTS_DIR"/*.csv; do
    [[ -f "$export_file" ]] || continue
    filename=$(basename "$export_file")
    wave_date=$(echo "$filename" | grep -oP '\d{8}')

    echo "Processing: $filename (wave: $wave_date)"

    # Validate: must have required columns
    header=$(head -1 "$export_file")
    for col in passion_point_id brand_id country_code alignment_index base_size; do
        echo "$header" | grep -q "$col" || { echo "MISSING COLUMN: $col in $filename"; continue 2; }
    done

    # Filter: only base_size >= 100
    awk -F',' 'NR==1 || $NF >= 100' "$export_file" > "$PROCESSED_DIR/${wave_date}_${filename}"

    # Archive original
    gzip -c "$export_file" > "$ARCHIVE_DIR/${filename}.gz"
    rm "$export_file"

    echo "✓ Processed: $filename → $PROCESSED_DIR/${wave_date}_${filename}"
done

Wave tracking data ingestion

# Ingest new wave CSV into database
ingest_wave() {
    local csv_file="$1"
    local wave_id="$2"
    local study_id="$3"

    # Row count validation
    row_count=$(awk 'NR>1' "$csv_file" | wc -l | tr -d ' ')
    [[ $row_count -lt 100 ]] && die "Wave CSV has only $row_count rows — suspiciously low"

    # Load to PostgreSQL
    psql "$IMI_DB_URL" <<SQL
        \COPY wave_responses(respondent_id, question_id, response_value, weight)
        FROM '$csv_file' CSV HEADER;
        UPDATE wave_responses SET wave_id = '$wave_id', study_id = '$study_id'
        WHERE wave_id IS NULL;
SQL

    echo "✓ Ingested $row_count rows for wave $wave_id / study $study_id"
}

Quick Reference: Essential Commands

*See also: python-data-engine (when shell + Python work together), data-pipeline (orchestrating multi-step data workflows), file-automation (complex file management patterns)*

AXE MCP Server Integration

Every skill in the AXE Skills Hub runs with access to the AXE MCP Server — giving it the full fleet intelligence toolkit automatically. No setup required; tools are available in any AXE-powered session.

Core Tools Available

Quick Start

# In any AXE session, tools are pre-loaded. Example chaining:

# 1. Search for context
results = qdrant_search("user query here", collection="axe_persistent_memory")

# 2. Fetch live data if needed
content = web_fetch("https://docs.example.com/api")

# 3. Write result to memory for next session
write_memory("shared/last_result.md", output)

# 4. Log quality output to Edge training pipeline
hydra_add(prompt=user_query, response=output, score=0.9, source="skill-name")

Edge Training Integration

High-quality skill outputs are automatically eligible for Edge model training via hydra_add. When a response scores ≥0.85 in evals, pipe it to the Hydra pipeline to compound Edge's knowledge. This is how skills make Edge smarter over time.

# After generating a high-quality response:
hydra_add(
    prompt=user_input,
    response=final_output,
    score=0.9,          # eval score
    source="skill-name" # tracks provenance
)