Flask App Organization

MVC isn’t perfect alone

It’s probably worth establishing a bit of foundational philosophy at the beginning of this series. Fair warning in advance that this will be a bit of a longer post.

MVC is undoubtedly a viable and smart approach to developing software. It can help you get started quickly without thinking too much about anything beyond models, views, and controllers. If you at least roughly know the expectations and goals of MVC, you can sort of coast your way through developing an application by just sticking all your code in one of the three buckets.

Unfortunately, MVC enables—and perhaps encourages—a developer to somewhat ignore thinking about application architecture and organization before getting started. In my opinion, MVC pretty much suggests thinking that your models are the center of your app, and your views are where all app functionality should happen. For a simple CRUD app, MVC is definitely the quickest path to creating a fairly sane and easy-to-understand codebase. However, when you begin moving past simple CRUD operations and are building anything of moderate complexity, strict adherence to the MVC pattern can result in pretty undesirable outcomes.

Note: For the purists out there, just a quick word to acknowledge that Python web apps—be they Django, Flask, or other frameworks—tend to actually be MVT, not MVC. That is, apps are typically organized with models, views, and templates. However, the communities around the various Python web frameworks still refer to this as MVC, and I’m not going to litigate the terminology here.

Domain-driven design feels natural

A couple of years ago, I started playing with Elixir—for curiosity and other reasons outside the scope of this series. That triggered taking a deeper look at how Elixir projects approached app organization, which led to studying various other frameworks, techniques, and patterns for ideas on organizing apps.

One approach that stood out to me—and has since become the default approach in Phoenix-based apps—was domain-driven design (DDD). I was working on some pretty complicated apps that were growing increasingly difficult to keep organized with a simple MVC approach—database objects were needed by multiple parts of the app, views were operating on multiple different types of models to get things done, and things were just getting pretty overwhelming in the code base.

As I started taking a higher-level look at my Flask apps, I recognized some of the core concepts of DDD somewhat jumping out at me as natural organizational opportunities. Sure, I had models and views in typical MVC fashion. However, I was already organizing those models and views somewhat intuitively along contextual lines. I was building various helper modules in my apps that tried to isolate certain workflows and behaviors to keep views as clean as possible. At the same time, without a concerted effort and a guiding principle, it was still easy to find too much logic nested inside views, models growing wildly long with properties and various behavior, workflow, and mutation methods, or helper functions and modules in a somewhat scattered state with no clear organizing principle helping them be intuitively discoverable.

At this time, I was in the middle of a project that had grown to around 200K lines of code. I have a particularly strong knack for keeping an instantly recallable map of software in my head to guide development (and rapidly identify bugs), but discovered that isn’t always the case for other developers. After bringing in a new junior developer to a project that had so far only been staffed with senior devs, I began recognizing it wasn’t easy to bring someone new up to speed on not only the codebase, but the core guiding principles and expecations that were in play. As I started working to help build this mental map in another developer’s mind, I gravitated toward explaining the app’s features and functionality along DDD lines—contexts, domains, entities, value objects, and so on (without using these terms explicitly so as not to introduce even more cognitive overhead). It was at this point that I really began to see the payoff in just talking about the application in a domain-driven fashion—successfully helping the new developer build up a mental map of the software—and decided to work on implementing that domain-driven map in the actual codebase.

This reorganization of a fairly complex application took a couple of months to complete. When it was done, however, the team succeeded in removing over 70K lines of code while expanding the available features, made the entire codebase intuitively navigable, and vastly increased the testability of the software by thinking about DDD-based organization and tests first.

Note: By tests first, I don’t mean TDD. TDD isn’t really my preferred approach to testing software. I’m the kind of person who prefers to start prototyping and building while thinking about the higher-level organization of the app as a whole. I begin writing tests once I feel like I’ve solidified the approach well enough to know it’s properly organized by its natural domains and bounded contexts. During this process, however, I’m always thinking about code writing and organization from the standpoint of how can this function, class, or module be written to be easily testable—so it’s sort of a future TDD approach. Maybe I should call that FTDD.

Let’s take a quick look at how I apply some DDD principles to a Flask app.

Models

Models are, of course, how you structure what data your app will persist to a storage backend. In nearly all web frameworks, a developer defines a model class—in Flask apps, that’s going to be an SQLAlchemy db.Model— then adds some attributes of various types to represent columns, and relies on an ORM (usually) to query, interact with, mutate, and persist objects in the database.

Beware of fat models

Trouble begins when your models need to move beyond simple CRUD-style operations that are typically provided by your ORM already—when you need custom attributes, computed properties, behaviors, complex queries, workflows, and a number of other state- and behavior-driven features. When approaching these kinds of issues from a strict MVC approach, it’s only a matter of time before a complex app will begin to see models that grow to hundreds of lines of code or more. In some of the more complex apps I’ve built and helped build over the last decade, I’ve watched as models grow fat and become difficult to navigate, and seen developers struggle to add and modify code in an organized fashion.

Keep models thin

One of the first things I did when I began applying a DDD-informed approach to organizing my Flask apps was decide that model definitions were only ever going to be concerned with database schema. Well, technically, they’re more than that thanks to leveraging mixins. However, when dealing with the codebase, a model file is only allowed to define its database schema—columns, indexes, relationships, and a custom save method if needed to perform extra work before persisting (such as updating related objects, or something similar).

Now, as the subtitle of this post says, I approach Flask app organization with a mix of MVC & domain-driven design. One place this is apparent is I still prefer to keep my models housed in a models package/folder. But where those models need properties, behaviors, mutation, workflows, and other non-database-focused code, that all lives inside a bounded context package that represents that portion of the app’s domain and functionality. Typically, this code is written as mixins and interface classes—some of which will be added to a model’s class definition—and the separation helps keep organization a first principle while developing. One nice benefit here is that each of these mixins are more easily tested—and isolated during tests—which helps ensure all the various building blocks of an application are considered as independent, testable pieces, instead of massive tests that are trying to cover massive models.

Thin model example

Here’s a quick look at how I approach models with a typical User model in one of my apps:

# proj/models/user.py

from proj.db import db
from proj.models import Model # more on this in another post
from proj.accounts.mixins.user import UserMixin


class User(Model, UserMixin):
    """App user model.

    We have to set the `__tablename__` attr to `users` or we conflict with PostgreSQL's
    internal `user` table.
    """
    __tablename__ = 'users'
    # internal attrs
    pk = db.Column(db.Integer, primary_key=True)
    active = db.Column(db.Boolean, default=True, nullable=False, index=True)
    # ... more model attrs ...

The UserMixin class itself tends to be built from a number of other mixins that all isolate certain responsibilities, properties, and behaviors that matter for the app itself. We may look at that further in the series in another post.

Views

A web framework wouldn’t be anything without a view layer, and Flask is no exception. In my Flask apps, I exclusively rely on Flask-Classful for class-based views, which I tend to customize in various ways to bring custom functionality, permission-enforcement, and other features to my app views.

Views can get fat, too

When following a strict MVC approach to web app development, it is all too easy to see views grow increasingly lengthy, complex, brittle, and difficult to reason about over time—especially as the app ages or features expand beyond original expectations.

One major danger inherent in letting views grow out of control is it becomes increasingly difficult to reason about how to implement new functionality—you wind up asking yourself if you should just add more arguments and conditionals to existing views, add new views, break up existing views into smaller pieces to try to handle the logical complexity, etc.

View-based testing is the wrong approach to prove code works

Perhaps more dangerous and difficult than dealing with organizing code when implementing features from a view-centric approach is realizing that building core functionality inside views becomes very unpleasant to test. Sure, Flask provides a relatively straightforward way to setup a request context for testing, but I’ve found the expectation that tests ought to be executed in the context of a request to prove correct app functionality to be quite a bizarre expectation. Yes, you want to be sure your requests perform as expected—but for me, that means proving a specific request returns an expected response. Views shouldn’t be responsible for mutating objects, updating data, or just about anything other than taking in a request, taking some action, and returning an expected response without an error.

One of the really unfortunate side effects of Flask being overly view-centric is that there are a ton of Flask extensions that are nearly impossible to extend and test as part of your application without a Request being present—the libraries themselves are actually built expecting to only ever exist in the context of a Request, which makes it quite challenging to implement functionality you want to prove is correct without a request context. I’m going to cover this in considerably more detail when the series tackles views and testing directly.

Keep views thin

DDD provides an excellent approach for handling organizing, implementing, and testing app code while building thin views. You build various bounded contexts for your app domain, implement interfaces through which you pass your entities and value objects—interfaces that can be tested and proven for correctness—and your views become responsible for routing requests through appropriate contexts. It also happens to make views increasingly more easy to reason about.

Thin view example

Here’s a look at how I would handle a complex set of operations on a Client object from inside a view using this approach to keep views thin, while building functionality that lives in its proper bounded context, and remains testable for correctness itself:

# proj/views/client.py

from proj.client import Client
from proj.views import AdminView

class ClientView(AdminView):
    # ... other views ...

    def complex(self, pk):
        obj = Client.get(pk)
        Client.complex_action(obj)
        obj.save()
        return html('some_template.html', client=obj)

    # ... more views ...

With this DDD-inspired approach, my view is only responsible for accepting a pk so Client.get can find the proper item in the database, passing that object to Client.complex_action, saving the object when the complex action is finished, and returning a response. My view doesn’t—and shouldn’t!—need to care what complex_action is doing. It could be anything at all. It’s not the view’s job to ensure complex_action does its job correctly. That’s what testing the Client interface is for—there will be tests for complex_action that prove it is correct. The view only needs to be concerned with accepting the Request, triggering the action expected to be performed, and returning a Response.

A sample app layout

It’s probably easiest at this point to close with what a Flask app looks like the way I combine an MVC- + DDD-driven approach to app organization. I’ll cover many of these pieces in more detail in future posts.

$ tree proj_root

proj_root/              # main dir holding everything for the project

  config/               # main config dir for app
    config.yml          # default config file

  app/                  # app package
    __init__.py         # where we find create_app()

    accounts/           # bounded context for user accounts
      __init__.py
      account.py        # holds Account interface for all user actions, behaviors, etc.
      mixins/
        __init.py
        user.py         # main mixin that will be added to models/user.py
        role.py         # holds role-base properties
        ability.py      # holds properties that identify what abilities a user has
        identity.py     # holds properties used to identify users
      queries.py        # holds custom queries that act on User objects

    clients/            # bounded contexts for clients
      client.py         # main Client interface
      mixins/
        __init__.py
        client.py       # main mixin that will be added to models/client.py

    db.py               # main database setup/init module

    models
      __init__.py
      client.py         # schema for client table
      user.py           # schema for users table

    templates/

    views
      __init__.py
      client.py         # view class for operating on clients
      user.py           # view class for operating on users

Up next

That was quite a lot to cover. Next, we’re going to look at the Flask app factory pattern.