For some reason eager-loaded data isn't passed to nested serializers when using associations. Defining the association as a method and passing in some optional paramaters resolves the issue.
This reduces queries run by the controller action by around 75%, as well as the response times.