Skip to main content

Managing Large File Uploads Securely in Flask with Flask-WTF

Managing Large File Uploads Securely in Flask with Flask-WTF

Managing large file uploads in Flask is critical for applications handling substantial datasets, such as CSV files for Pandas processing or inputs for machine learning (ML) models. Large files introduce challenges like server resource exhaustion, long upload times, and increased security risks (e.g., denial-of-service attacks, path traversal, and CSRF). Flask-WTF, combined with best practices, provides a robust framework for securely handling large file uploads. This guide covers managing large file uploads securely in Flask, including setup, validation, chunked uploads, secure storage, and practical examples, with a focus on data-driven applications.

01. Overview of Large File Uploads

Large file uploads require careful handling to ensure server stability, security, and user experience. Flask-WTF facilitates secure uploads with validation and CSRF protection, while additional techniques like chunked uploads and asynchronous processing can optimize performance for large files.

  • Purpose: Enable users to upload large files securely while maintaining server performance.
  • Key Components: FileField, FileAllowed, secure_filename, size limits, and chunked uploads.
  • Use Cases: Uploading large CSVs for data analysis, ML model training datasets, or multimedia files.

1.1 Challenges and Risks

  • Resource Exhaustion: Large files can consume memory and disk space, leading to crashes.
  • Path Traversal: Malicious filenames can access unauthorized directories.
  • CSRF Attacks: Unauthorized submissions can exploit authenticated sessions.
  • Timeout Issues: Long uploads may exceed server timeouts.
  • Malicious Files: Large files may hide malicious content (e.g., executables).

02. Setting Up Flask-WTF for Large File Uploads

2.1 Installation

Install Flask-WTF and dependencies: 

pip install flask-wtf

2.2 Project Structure

project/
├── app.py
├── static/
│   ├── css/
│   │   └── style.css
│   ├── js/
│   │   └── upload.js
├── templates/
│   ├── base.html
│   ├── upload.html
│   └── result.html
└── uploads/

2.3 Basic Configuration

Configure Flask with a secret key for CSRF protection, an upload folder, and a maximum file size limit.

Example: Basic Large File Upload

File: app.py

from flask import Flask, render_template
from flask_wtf import FlaskForm
from flask_wtf.file import FileField, FileAllowed, FileRequired
from wtforms import SubmitField
from werkzeug.utils import secure_filename
import os

app = Flask(__name__)
app.config['SECRET_KEY'] = 'your-secret-key'
app.config['UPLOAD_FOLDER'] = 'uploads'
app.config['MAX_CONTENT_LENGTH'] = 100 * 1024 * 1024  # 100MB limit

class UploadForm(FlaskForm):
    file = FileField('CSV File', validators=[FileRequired(), FileAllowed(['csv'], 'CSV files only')])
    submit = SubmitField('Upload')

@app.errorhandler(413)
def too_large(e):
    return render_template('error.html', error='File too large (max 100MB)'), 413

@app.route('/upload', methods=['GET', 'POST'])
def upload():
    form = UploadForm()
    if form.validate_on_submit():
        file = form.file.data
        filename = secure_filename(file.filename)
        file_path = os.path.join(app.config['UPLOAD_FOLDER'], filename)
        file.save(file_path)
        return render_template('result.html', filename=filename)
    return render_template('upload.html', form=form)

if __name__ == '__main__':
    os.makedirs('uploads', exist_ok=True)
    os.chmod('uploads', 0o700)  # Restrictive permissions
    app.run(debug=True)

File: templates/base.html

<!DOCTYPE html>
<html>
<head>
    <title>{% block title %}Flask Large File Upload{% endblock %}</title>
    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.5.2/css/bootstrap.min.css">
    <link rel="stylesheet" href="{{ url_for('static', filename='css/style.css') }}">
</head>
<body>
    <div class="container mt-3">
        {% block content %}{% endblock %}
    </div>
</body>
</html>

File: static/css/style.css

body {
    font-family: Arial, sans-serif;
}
.form-group {
    margin-bottom: 15px;
}
.error {
    color: red;
    font-size: 0.9em;
}

File: templates/upload.html

{% extends 'base.html' %}

{% block title %}Upload File{% endblock %}

{% block content %}
    <h1>Upload CSV File (Max 100MB)</h1>
    <form method="post" enctype="multipart/form-data" novalidate>
        {{ form.hidden_tag() }}
        <div class="form-group">
            {{ form.file.label }}
            {{ form.file(class="form-control") }}
            {% for error in form.file.errors %}
                <span class="error">{{ error | escape }}</span>
            {% endfor %}
        </div>
        {{ form.submit(class="btn btn-primary") }}
    </form>
{% endblock %}

File: templates/result.html

{% extends 'base.html' %}

{% block title %}Upload Success{% endblock %}

{% block content %}
    <h1>File Uploaded</h1>
    <p>File: {{ filename | escape }}</p>
    <a href="{{ url_for('upload') }}" class="btn btn-secondary">Upload Another</a>
{% endblock %}

File: templates/error.html

{% extends 'base.html' %}

{% block title %}Error{% endblock %}

{% block content %}
    <h1>Error</h1>
    <p>{{ error | escape }}</p>
    <a href="{{ url_for('upload') }}" class="btn btn-secondary">Try Again</a>
{% endblock %}

Output (/upload):

  • GET: Displays a Bootstrap-styled file upload form.
  • POST (valid CSV < 100MB): Saves the file and shows a success page.
  • POST (non-CSV): Shows "CSV files only."
  • POST (file > 100MB): Shows "File too large (max 100MB)."

Explanation:

  • MAX_CONTENT_LENGTH: Limits uploads to 100MB.
  • secure_filename: Sanitizes filenames to prevent path traversal.
  • FileAllowed(['csv']): Restricts to CSV files.
  • @app.errorhandler(413): Handles oversized file errors.
  • form.hidden_tag(): Ensures CSRF protection.

03. Strategies for Managing Large File Uploads

3.1 File Size Limits

Use MAX_CONTENT_LENGTH to restrict upload size and handle 413 errors gracefully.

3.2 Chunked Uploads

For very large files, chunked uploads split files into smaller pieces, reducing memory usage and improving reliability. This requires client-side JavaScript and server-side logic to reassemble chunks.

3.3 Asynchronous Processing

Process large files asynchronously (e.g., using Celery) to avoid blocking the server during validation or processing.

3.4 Validation and Security

  • File Type: Use FileAllowed to restrict extensions.
  • Content Validation: Verify file contents (e.g., CSV structure with Pandas).
  • Filename Sanitization: Use secure_filename to prevent path traversal.
  • CSRF Protection: Include form.hidden_tag() for security.

3.5 Storage Optimization

  • Store files in a dedicated folder outside the web root.
  • Use unique filenames (e.g., with UUIDs) to prevent overwrites.
  • Set restrictive permissions (e.g., 0o700).

04. Implementing Chunked File Uploads

Chunked uploads are ideal for very large files, allowing clients to send files in smaller parts, which the server reassembles.

Example: Chunked File Upload with JavaScript

File: app.py

from flask import Flask, jsonify, request, render_template
from flask_wtf.csrf import CSRFProtect
from werkzeug.utils import secure_filename
import os
import uuid

app = Flask(__name__)
app.config['SECRET_KEY'] = 'your-secret-key'
app.config['UPLOAD_FOLDER'] = 'uploads'
app.config['MAX_CONTENT_LENGTH'] = 100 * 1024 * 1024  # 100MB limit
csrf = CSRFProtect(app)

@app.errorhandler(413)
def too_large(e):
    return jsonify({'error': 'File too large (max 100MB)'}), 413

@app.route('/chunked-upload', methods=['POST'])
@csrf.exempt  # CSRF handled in JavaScript
def chunked_upload():
    chunk = request.files['chunk']
    filename = secure_filename(request.form['filename'])
    chunk_index = int(request.form['chunkIndex'])
    total_chunks = int(request.form['totalChunks'])
    unique_id = request.form['uniqueId']

    temp_dir = os.path.join(app.config['UPLOAD_FOLDER'], unique_id)
    os.makedirs(temp_dir, exist_ok=True)

    chunk_path = os.path.join(temp_dir, f'chunk_{chunk_index}')
    chunk.save(chunk_path)

    # Check if all chunks are uploaded
    if chunk_index + 1 == total_chunks:
        final_path = os.path.join(app.config['UPLOAD_FOLDER'], f"{unique_id}_{filename}")
        with open(final_path, 'wb') as final_file:
            for i in range(total_chunks):
                with open(os.path.join(temp_dir, f'chunk_{i}'), 'rb') as chunk_file:
                    final_file.write(chunk_file.read())
                os.remove(os.path.join(temp_dir, f'chunk_{i}'))
        os.rmdir(temp_dir)
        return jsonify({'status': 'success', 'filename': f"{unique_id}_{filename}"})
    
    return jsonify({'status': 'chunk_received'})

@app.route('/upload', methods=['GET'])
def upload():
    return render_template('upload.html')

if __name__ == '__main__':
    os.makedirs('uploads', exist_ok=True)
    os.chmod('uploads', 0o700)
    app.run(debug=True)

File: templates/upload.html

{% extends 'base.html' %}

{% block title %}Chunked File Upload{% endblock %}

{% block content %}
    <h1>Upload Large CSV File (Max 100MB)</h1>
    <form id="uploadForm" enctype="multipart/form-data">
        <div class="form-group">
            <label for="file">CSV File:</label>
            <input type="file" id="file" class="form-control" accept=".csv" required>
        </div>
        <button type="submit" class="btn btn-primary">Upload</button>
    </form>
    <p id="status"></p>
    <script src="{{ url_for('static', filename='js/upload.js') }}"></script>
{% endblock %}

File: static/js/upload.js

document.getElementById('uploadForm').addEventListener('submit', async (e) => {
    e.preventDefault();
    const fileInput = document.getElementById('file');
    const file = fileInput.files[0];
    const status = document.getElementById('status');

    if (!file.name.endsWith('.csv')) {
        status.textContent = 'Only CSV files are allowed.';
        return;
    }

    const chunkSize = 5 * 1024 * 1024; // 5MB chunks
    const totalChunks = Math.ceil(file.size / chunkSize);
    const uniqueId = crypto.randomUUID();
    const filename = file.name;

    // Fetch CSRF token
    const csrfResponse = await fetch('/get-csrf-token');
    const { csrf_token } = await csrfResponse.json();

    for (let i = 0; i < totalChunks; i++) {
        const start = i * chunkSize;
        const end = Math.min(start + chunkSize, file.size);
        const chunk = file.slice(start, end);

        const formData = new FormData();
        formData.append('chunk', chunk);
        formData.append('filename', filename);
        formData.append('chunkIndex', i);
        formData.append('totalChunks', totalChunks);
        formData.append('uniqueId', uniqueId);

        try {
            const response = await fetch('/chunked-upload', {
                method: 'POST',
                headers: {
                    'X-CSRF-Token': csrf_token
                },
                body: formData
            });
            const result = await response.json();
            status.textContent = `Uploaded chunk ${i + 1}/${totalChunks}`;

            if (result.status === 'success') {
                status.textContent = `File uploaded: ${result.filename}`;
            }
        } catch (error) {
            status.textContent = 'Upload failed: ' + error.message;
            return;
        }
    }
});

// Fetch CSRF token endpoint
app.route('/get-csrf-token', methods=['GET'])
def get_csrf_token():
    from flask_wtf.csrf import generate_csrf
    return jsonify({'csrf_token': generate_csrf()})

Output (/upload):

  • GET: Displays a file upload form.
  • POST (valid CSV): Uploads in 5MB chunks, reassembles, and shows success.
  • POST (non-CSV): Client-side check shows "Only CSV files are allowed."
  • POST (file > 100MB): Server returns "File too large (max 100MB)."

Explanation:

  • Chunked Upload: Splits files into 5MB chunks for efficient handling.
  • secure_filename: Sanitizes filenames.
  • uuid: Ensures unique temporary directories.
  • CSRF: Handled via X-CSRF-Token header in JavaScript.
  • @csrf.exempt: Allows manual CSRF handling for the chunked endpoint.

05. Managing Large File Uploads in Data-Driven Applications

Large file uploads are common in data-driven applications, such as uploading datasets for Pandas analysis or ML training.

Example: Large CSV Upload with Pandas Validation

File: app.py

from flask import Flask, render_template
from flask_wtf import FlaskForm
from flask_wtf.file import FileField, FileAllowed, FileRequired
from wtforms import SubmitField
from werkzeug.utils import secure_filename
import pandas as pd
import os
import uuid

app = Flask(__name__)
app.config['SECRET_KEY'] = 'your-secret-key'
app.config['UPLOAD_FOLDER'] = 'uploads'
app.config['MAX_CONTENT_LENGTH'] = 100 * 1024 * 1024  # 100MB limit

class UploadForm(FlaskForm):
    file = FileField('CSV File', validators=[FileRequired(), FileAllowed(['csv'], 'CSV files only')])
    submit = SubmitField('Upload')

@app.errorhandler(413)
def too_large(e):
    return render_template('error.html', error='File too large (max 100MB)'), 413

@app.route('/upload', methods=['GET', 'POST'])
def upload():
    form = UploadForm()
    if form.validate_on_submit():
        file = form.file.data
        unique_id = str(uuid.uuid4())
        filename = secure_filename(f"{unique_id}_{file.filename}")
        file_path = os.path.join(app.config['UPLOAD_FOLDER'], filename)
        
        # Save file
        file.save(file_path)
        
        # Validate CSV with Pandas
        try:
            df = pd.read_csv(file_path)
            if df.empty:
                return render_template('upload.html', form=form, error='Empty CSV file')
            expected_columns = ['Name', 'Age']  # Example
            if not all(col in df.columns for col in expected_columns):
                return render_template('upload.html', form=form, error='Missing required columns')
            return render_template('result.html', filename=filename, data=df.to_dict(orient='records'))
        except Exception as e:
            return render_template('upload.html', form=form, error=f'Invalid CSV: {str(e)}')
    
    return render_template('upload.html', form=form)

if __name__ == '__main__':
    os.makedirs('uploads', exist_ok=True)
    os.chmod('uploads', 0o700)
    app.run(debug=True)

File: templates/upload.html

{% extends 'base.html' %}

{% block title %}Upload CSV{% endblock %}

{% block content %}
    <h1>Upload Large CSV File (Max 100MB)</h1>
    {% if error %}
        <p class="text-danger">{{ error | escape }}</p>
    {% endif %}
    <form method="post" enctype="multipart/form-data" novalidate>
        {{ form.hidden_tag() }}
        <div class="form-group">
            {{ form.file.label }}
            {{ form.file(class="form-control") }}
            {% for error in form.file.errors %}
                <span class="error">{{ error | escape }}</span>
            {% endfor %}
        </div>
        {{ form.submit(class="btn btn-primary") }}
    </form>
{% endblock %}

File: templates/result.html

{% extends 'base.html' %}

{% block title %}Upload Success{% endblock %}

{% block content %}
    <h1>File Uploaded</h1>
    <p>File: {{ filename | escape }}</p>
    <h2>Data Preview</h2>
    <table class="table table-striped">
        <thead>
            <tr>
                {% for key in data[0].keys() %}
                    <th>{{ key | title }}</th>
                {% endfor %}
            </tr>
        </thead>
        <tbody>
            {% for row in data %}
                <tr>
                    {% for value in row.values() %}
                        <td>{{ value | escape }}</td>
                    {% endfor %}
                </tr>
            {% endfor %}
        </tbody>
    </table>
    <a href="{{ url_for('upload') }}" class="btn btn-secondary">Upload Another</a>
{% endblock %}

Output (/upload):

  • GET: Displays a file upload form.
  • POST (valid CSV < 100MB): Saves the file, validates it, and shows a data preview.
  • POST (invalid CSV): Shows errors like "Invalid CSV: ..." or "Missing required columns."
  • POST (file > 100MB): Shows "File too large (max 100MB)."

Explanation:

  • Pandas: Validates CSV structure and content.
  • uuid: Ensures unique filenames.
  • secure_filename: Sanitizes filenames.
  • CSRF: Protected via form.hidden_tag().

06. Best Practices for Managing Large File Uploads

6.1 Recommended Practices

  • Limit File Size: Set MAX_CONTENT_LENGTH and handle 413 errors.
  • Use Chunked Uploads: Split large files into smaller chunks for efficiency.
  • Validate File Types: Use FileAllowed and check MIME types.
  • Validate Content: Verify file contents (e.g., with Pandas for CSVs).
  • Sanitize Filenames: Use secure_filename.
  • Use Unique Filenames: Append UUIDs or timestamps.
  • Secure Storage: Store files outside the web root with restrictive permissions.
  • Enable CSRF Protection: Use form.hidden_tag() or manual CSRF tokens for chunked uploads.
  • Asynchronous Processing: Use Celery for long-running tasks.

6.2 Security Considerations

  • Antivirus Scanning: Integrate tools like ClamAV in production.
  • Timeout Handling: Configure server timeouts to handle long uploads.
  • Secure Outputs: Use | escape for filenames in templates.
  • Logging: Log upload attempts for auditing.

Example: Insecure Large File Upload

File: app.py

from flask import Flask, request
import os

app = Flask(__name__)
app.config['UPLOAD_FOLDER'] = 'uploads'

@app.route('/insecure', methods=['POST'])
def insecure():
    file = request.files['file']
    file.save(os.path.join(app.config['UPLOAD_FOLDER'], file.filename))  # Insecure
    return 'File uploaded'

Issues:

  • No CSRF protection.
  • No filename sanitization.
  • No file type or size validation.
  • No content validation.

Correct: Use Flask-WTF, secure_filename, and chunked uploads as shown earlier.

Explanation:

  • Insecure: Vulnerable to malicious files, path traversal, and resource exhaustion.
  • Correct: Secure practices mitigate these risks.

6.3 Practices to Avoid

  • Avoid Raw request.files: Use Flask-WTF for validation.
  • Avoid Unlimited Sizes: Always set MAX_CONTENT_LENGTH.
  • Avoid Blocking Uploads: Use chunked or asynchronous uploads for large files.

07. Conclusion

Managing large file uploads securely in Flask is essential for data-driven applications. Key takeaways:

  • Use Flask-WTF with FileField and validators for secure uploads.
  • Implement chunked uploads for very large files to improve performance.
  • Validate file types, sizes, and contents, and sanitize filenames.
  • Ensure CSRF protection and store files securely outside the web root.

By adopting these practices, you can build robust Flask applications that safely handle large file uploads, supporting workflows like data analysis and ML with confidence!

Comments

Post a Comment