Flask: Middleware for Logging and Metrics

Middleware in Flask is a powerful way to centralize logging and metrics collection, enabling developers to monitor application behavior, track performance, and debug issues efficiently. By leveraging WSGI middleware, Flask applications can capture request details, measure latency, and integrate with tools like Prometheus for metrics. This tutorial explores Flask middleware for logging and metrics, covering setup, implementation, integration with Prometheus, and best practices for robust monitoring.

01. Why Use Middleware for Logging and Metrics in Flask?

Middleware allows global processing of requests and responses, making it ideal for logging request details (e.g., method, path, client IP) and collecting metrics (e.g., latency, request counts). Flask’s WSGI-based architecture supports middleware to centralize these concerns, reducing code duplication and improving maintainability. Logging and metrics ensure observability, performance optimization, and proactive issue detection, critical for production-grade applications.

Example: Basic Logging Middleware

from flask import Flask
from werkzeug.wrappers import Request
import logging

app = Flask(__name__)

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class LoggingMiddleware:
    def __init__(self, wsgi_app):
        self.wsgi_app = wsgi_app

    def __call__(self, environ, start_response):
        request = Request(environ)
        logger.info(f"{request.method} {request.path} from {request.remote_addr}")
        return self.wsgi_app(environ, start_response)

app.wsgi_app = LoggingMiddleware(app.wsgi_app)

@app.route('/')
def index():
    return "Hello, Flask!"

if __name__ == '__main__':
    app.run(debug=True, port=5000)

Output:

* Running on http://127.0.0.1:5000
INFO:__main__:GET / from 127.0.0.1

Explanation:

LoggingMiddleware - Logs request method, path, and client IP for all routes.
app.wsgi_app - Integrates middleware into Flask’s WSGI pipeline.

02. Key Middleware Techniques for Logging and Metrics

Middleware for logging and metrics can capture detailed request data, measure performance, and integrate with monitoring systems. These techniques ensure comprehensive observability. The table below summarizes key techniques and their applications:

Technique	Description	Use Case
Request Logging	Log request details (method, path, IP)	Auditing, debugging
File-Based Logging	Save logs to files	Persistent monitoring
Latency Metrics	Measure request processing time	Performance optimization
Prometheus Metrics	Expose metrics for scraping	Real-time monitoring
Error Tracking	Log exceptions and errors	Issue diagnosis

2.1 Request Logging Middleware

Example: Detailed Request Logging

from flask import Flask
from werkzeug.wrappers import Request
import logging

app = Flask(__name__)

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class DetailedLoggingMiddleware:
    def __init__(self, wsgi_app):
        self.wsgi_app = wsgi_app

    def __call__(self, environ, start_response):
        request = Request(environ)
        logger.info(
            f"Request: {request.method} {request.path}, "
            f"IP: {request.remote_addr}, User-Agent: {request.user_agent}"
        )
        return self.wsgi_app(environ, start_response)

app.wsgi_app = DetailedLoggingMiddleware(app.wsgi_app)

@app.route('/')
def index():
    return "Detailed Logging"

if __name__ == '__main__':
    app.run(debug=True, port=5000)

Output:

* Running on http://127.0.0.1:5000
INFO:__main__:Request: GET /, IP: 127.0.0.1, User-Agent: Mozilla/5.0 ...

Explanation:

Logs detailed request info, including method, path, IP, and User-Agent.
Useful for auditing and debugging.

2.2 File-Based Logging Middleware

Example: Logging to File

from flask import Flask
from werkzeug.wrappers import Request
import logging
from logging.handlers import RotatingFileHandler

app = Flask(__name__)

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
handler = RotatingFileHandler('app.log', maxBytes=1000000, backupCount=5)
handler.setFormatter(logging.Formatter('%(asctime)s - %(message)s'))
logger.addHandler(handler)

class FileLoggingMiddleware:
    def __init__(self, wsgi_app):
        self.wsgi_app = wsgi_app

    def __call__(self, environ, start_response):
        request = Request(environ)
        logger.info(f"{request.method} {request.path} from {request.remote_addr}")
        return self.wsgi_app(environ, start_response)

app.wsgi_app = FileLoggingMiddleware(app.wsgi_app)

@app.route('/')
def index():
    return "Logged to File"

if __name__ == '__main__':
    app.run(debug=True, port=5000)

Output (app.log):

2025-05-12 10:00:00,123 - GET / from 127.0.0.1

Explanation:

RotatingFileHandler - Saves logs to a file with rotation to manage size.
Ensures persistent logging for later analysis.

2.3 Latency Metrics Middleware

Example: Measuring Request Latency

from flask import Flask
from werkzeug.wrappers import Request
import logging
import time

app = Flask(__name__)

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class LatencyMiddleware:
    def __init__(self, wsgi_app):
        self.wsgi_app = wsgi_app

    def __call__(self, environ, start_response):
        request = Request(environ)
        start_time = time.time()
        response = self.wsgi_app(environ, start_response)
        latency = time.time() - start_time
        logger.info(f"Latency: {latency:.3f}s for {request.method} {request.path}")
        return response

app.wsgi_app = LatencyMiddleware(app.wsgi_app)

@app.route('/')
def index():
    return "Latency Measured"

if __name__ == '__main__':
    app.run(debug=True, port=5000)

Output:

* Running on http://127.0.0.1:5000
INFO:__main__:Latency: 0.002s for GET /

Explanation:

Measures and logs request latency for performance monitoring.
Helps identify slow endpoints.

2.4 Prometheus Metrics Middleware

Example: Prometheus Metrics Collection

from flask import Flask
from werkzeug.wrappers import Request
from prometheus_client import Counter, Histogram, make_wsgi_app
from werkzeug.middleware.dispatcher import DispatcherMiddleware
import time

app = Flask(__name__)

# Define Prometheus metrics
request_count = Counter('flask_requests_total', 'Total requests', ['method', 'path', 'status'])
request_latency = Histogram('flask_request_latency_seconds', 'Request latency', ['path'])

class MetricsMiddleware:
    def __init__(self, wsgi_app):
        self.wsgi_app = wsgi_app

    def __call__(self, environ, start_response):
        request = Request(environ)
        start_time = time.time()

        # Capture response status
        status = [None]
        def custom_start_response(status_code, headers, *args):
            status[0] = status_code.split(' ')[0]  # Extract status code (e.g., 200)
            return start_response(status_code, headers, *args)

        response = self.wsgi_app(environ, custom_start_response)
        latency = time.time() - start_time

        # Record metrics
        request_count.labels(method=request.method, path=request.path, status=status[0]).inc()
        request_latency.labels(path=request.path).observe(latency)

        return response

app.wsgi_app = DispatcherMiddleware(MetricsMiddleware(app.wsgi_app), {
    '/metrics': make_wsgi_app()
})

@app.route('/')
def index():
    return "Metrics Tracked"

@app.route('/error')
def error():
    return "Error", 500

if __name__ == '__main__':
    app.run(debug=True, port=5000)

Output (http://127.0.0.1:5000/metrics):

flask_requests_total{method="GET",path="/",status="200"} 1.0
flask_request_latency_seconds_sum{path="/"} 0.002
flask_requests_total{method="GET",path="/error",status="500"} 1.0

Explanation:

Tracks request counts and latency with Prometheus metrics.
DispatcherMiddleware exposes metrics at /metrics for Prometheus scraping.

2.5 Error Tracking Middleware

Example: Logging Errors

from flask import Flask
from werkzeug.wrappers import Request, Response
import logging

app = Flask(__name__)

logging.basicConfig(level=logging.ERROR)
logger = logging.getLogger(__name__)

class ErrorLoggingMiddleware:
    def __init__(self, wsgi_app):
        self.wsgi_app = wsgi_app

    def __call__(self, environ, start_response):
        request = Request(environ)
        try:
            return self.wsgi_app(environ, start_response)
        except Exception as e:
            logger.error(f"Error in {request.method} {request.path}: {str(e)}", exc_info=True)
            res = Response('Internal Server Error', status=500)
            return res(environ, start_response)

app.wsgi_app = ErrorLoggingMiddleware(app.wsgi_app)

@app.route('/error')
def error():
    raise ValueError("Test error")

if __name__ == '__main__':
    app.run(debug=True, port=5000)

Output:

* Running on http://127.0.0.1:5000
ERROR:__main__:Error in GET /error: Test error
(Request to /error: Returns 500 Internal Server Error)

Explanation:

Catches and logs exceptions with stack traces.
Provides user-friendly error responses.

2.6 Incorrect Middleware Setup

Example: Missing Metrics Endpoint

from flask import Flask
from prometheus_client import Counter
from werkzeug.wrappers import Request

app = Flask(__name__)

request_count = Counter('flask_requests_total', 'Total requests', ['path'])

class MetricsMiddleware:
    def __init__(self, wsgi_app):
        self.wsgi_app = wsgi_app

    def __call__(self, environ, start_response):
        request = Request(environ)
        request_count.labels(path=request.path).inc()
        return self.wsgi_app(environ, start_response)

app.wsgi_app = MetricsMiddleware(app.wsgi_app)

@app.route('/')
def index():
    return "No Metrics Endpoint"

if __name__ == '__main__':
    app.run(debug=True, port=5000)

Output:

* Running on http://127.0.0.1:5000
(No /metrics endpoint; Prometheus cannot scrape metrics)

Explanation:

Missing make_wsgi_app prevents Prometheus metrics exposure.
Solution: Use DispatcherMiddleware to add /metrics.

03. Effective Usage

3.1 Recommended Practices

Combine logging and metrics middleware with file rotation and Prometheus for comprehensive monitoring.

Example: Comprehensive Logging and Metrics Middleware

from flask import Flask
from werkzeug.wrappers import Request, Response
from prometheus_client import Counter, Histogram, make_wsgi_app
from werkzeug.middleware.dispatcher import DispatcherMiddleware
import logging
from logging.handlers import RotatingFileHandler
import time

app = Flask(__name__)

# Setup logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
handler = RotatingFileHandler('app.log', maxBytes=1000000, backupCount=5)
handler.setFormatter(logging.Formatter('%(asctime)s - %(levelname)s - %(message)s'))
logger.addHandler(handler)

# Define Prometheus metrics
request_count = Counter('flask_requests_total', 'Total requests', ['method', 'path', 'status'])
request_latency = Histogram('flask_request_latency_seconds', 'Request latency', ['path'])

class LoggingAndMetricsMiddleware:
    def __init__(self, wsgi_app):
        self.wsgi_app = wsgi_app

    def __call__(self, environ, start_response):
        request = Request(environ)
        start_time = time.time()

        # Log request
        logger.info(f"{request.method} {request.path} from {request.remote_addr}")

        # Capture response status
        status = [None]
        def custom_start_response(status_code, headers, *args):
            status[0] = status_code.split(' ')[0]
            return start_response(status_code, headers, *args)

        try:
            response = self.wsgi_app(environ, custom_start_response)
            latency = time.time() - start_time

            # Record metrics
            request_count.labels(method=request.method, path=request.path, status=status[0]).inc()
            request_latency.labels(path=request.path).observe(latency)

            return response
        except Exception as e:
            logger.error(f"Error in {request.method} {request.path}: {str(e)}", exc_info=True)
            res = Response('Internal Server Error', status=500)
            return res(environ, start_response)

app.wsgi_app = DispatcherMiddleware(LoggingAndMetricsMiddleware(app.wsgi_app), {
    '/metrics': make_wsgi_app()
})

@app.route('/')
def index():
    return "Logged and Monitored"

@app.route('/error')
def error():
    raise ValueError("Test error")

if __name__ == '__main__':
    app.run(debug=True, port=5000)

Output (app.log):

2025-05-12 10:00:00,123 - INFO - GET / from 127.0.0.1
2025-05-12 10:00:01,456 - ERROR - Error in GET /error: Test error

Output (http://127.0.0.1:5000/metrics):

flask_requests_total{method="GET",path="/",status="200"} 1.0
flask_request_latency_seconds_sum{path="/"} 0.002
flask_requests_total{method="GET",path="/error",status="500"} 1.0

Combines logging (console and file) with Prometheus metrics.
Handles errors and exposes metrics for monitoring.
Uses file rotation to manage log size.

3.2 Practices to Avoid

Avoid excessive logging or metrics collection to prevent performance overhead.

Example: Overloaded Logging and Metrics

from flask import Flask
from werkzeug.wrappers import Request
from prometheus_client import Counter
import logging

app = Flask(__name__)

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

request_count = Counter('flask_requests_total', 'Total requests', ['method', 'path', 'ip', 'agent'])

class OverloadedMiddleware:
    def __init__(self, wsgi_app):
        self.wsgi_app = wsgi_app

    def __call__(self, environ, start_response):
        request = Request(environ)
        logger.debug(f"Request: {request.method} {request.path} {request.remote_addr} {request.user_agent}")
        request_count.labels(
            method=request.method, path=request.path,
            ip=request.remote_addr, agent=str(request.user_agent)
        ).inc()
        return self.wsgi_app(environ, start_response)

app.wsgi_app = OverloadedMiddleware(app.wsgi_app)

@app.route('/')
def index():
    return "Overloaded Logging"

if __name__ == '__main__':
    app.run(debug=True, port=5000)

Output:

* Running on http://127.0.0.1:5000
DEBUG:__main__:Request: GET / 127.0.0.1 Mozilla/5.0 ...
(Excessive logs and metrics slow down app)

DEBUG logging and too many metric labels cause overhead.
Solution: Use INFO level and limit metric labels (e.g., method, path).

04. Common Use Cases

4.1 API Request Monitoring

Log and monitor API requests for performance and usage insights.

Example: API Monitoring Middleware

from flask import Flask, jsonify
from werkzeug.wrappers import Request
from prometheus_client import Counter, Histogram, make_wsgi_app
from werkzeug.middleware.dispatcher import DispatcherMiddleware
import logging
import time

app = Flask(__name__)

logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter('%(asctime)s - %(message)s'))
logger.addHandler(handler)

request_count = Counter('api_requests_total', 'Total API requests', ['method', 'path'])
request_latency = Histogram('api_request_latency_seconds', 'API latency', ['path'])

class APIMonitoringMiddleware:
    def __init__(self, wsgi_app):
        self.wsgi_app = wsgi_app

    def __call__(self, environ, start_response):
        request = Request(environ)
        if not request.path.startswith('/api/'):
            return self.wsgi_app(environ, start_response)

        start_time = time.time()
        logger.info(f"API Request: {request.method} {request.path}")

        status = [None]
        def custom_start_response(status_code, headers, *args):
            status[0] = status_code.split(' ')[0]
            return start_response(status_code, headers, *args)

        response = self.wsgi_app(environ, custom_start_response)
        latency = time.time() - start_time

        request_count.labels(method=request.method, path=request.path).inc()
        request_latency.labels(path=request.path).observe(latency)

        return response

app.wsgi_app = DispatcherMiddleware(APIMonitoringMiddleware(app.wsgi_app), {
    '/metrics': make_wsgi_app()
})

@app.route('/api/data')
def data():
    return jsonify({'data': 'secure'})

if __name__ == '__main__':
    app.run(debug=True, port=5000)

Output (Console):

2025-05-12 10:00:00,123 - API Request: GET /api/data

Output (http://127.0.0.1:5000/metrics):

api_requests_total{method="GET",path="/api/data"} 1.0
api_request_latency_seconds_sum{path="/api/data"} 0.003

Explanation:

Logs and tracks metrics for API routes only.
Provides insights into API performance and usage.

4.2 Error Monitoring

Log and track errors for debugging and reliability.

Example: Error Monitoring Middleware

from flask import Flask
from werkzeug.wrappers import Request, Response
from prometheus_client import Counter, make_wsgi_app
from werkzeug.middleware.dispatcher import DispatcherMiddleware
import logging

app = Flask(__name__)

logger = logging.getLogger(__name__)
logger.setLevel(logging.ERROR)
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter('%(asctime)s - %(message)s'))
logger.addHandler(handler)

error_count = Counter('flask_errors_total', 'Total errors', ['path'])

class ErrorMonitoringMiddleware:
    def __init__(self, wsgi_app):
        self.wsgi_app = wsgi_app

    def __call__(self, environ, start_response):
        request = Request(environ)
        try:
            return self.wsgi_app(environ, start_response)
        except Exception as e:
            logger.error(f"Error in {request.method} {request.path}: {str(e)}", exc_info=True)
            error_count.labels(path=request.path).inc()
            res = Response('Internal Server Error', status=500)
            return res(environ, start_response)

app.wsgi_app = DispatcherMiddleware(ErrorMonitoringMiddleware(app.wsgi_app), {
    '/metrics': make_wsgi_app()
})

@app.route('/error')
def error():
    raise ValueError("Test error")

if __name__ == '__main__':
    app.run(debug=True, port=5000)

Output (Console):

2025-05-12 10:00:00,123 - Error in GET /error: Test error

Output (http://127.0.0.1:5000/metrics):

flask_errors_total{path="/error"} 1.0

Explanation:

Logs errors and tracks error counts with Prometheus.
Facilitates debugging and error rate analysis.

Conclusion

Middleware for logging and metrics in Flask enhances observability and performance monitoring. Key takeaways:

Use middleware to centralize logging (console/file) and metrics collection.
Integrate with Prometheus for real-time metrics and monitoring.
Implement error tracking for robust debugging.
Avoid excessive logging or metrics to maintain performance.

With these practices, you can build Flask applications with comprehensive logging and metrics for reliable operation and optimization!