Blending ORM and ODM but keeping the Domain Model clean

[webdevilopers]

When trying to use Doctrine ORM Entities with ODM Documents (e.g. MongoDB) most solutions adding extra fields:

• http://doctrine-orm.readthedocs.org/projects/doctrine-mongodb-odm/en/latest/cookbook/blending-orm-and-mongodb-odm.html @jmikola
• http://de.slideshare.net/spf13/blending-mongodb-and-rdbms-for-ecommerce by @jwage
• Add XML mapping example for References doctrine-extensions/DoctrineExtensions#1564 (comment)

For me this feels like "polluting" the Domain Model with extra properties to fit the infrastructure.
So I am currently looking for a way to keep the Domain Models clean:

• ODM double properties for ORM mapping doctrine-extensions/DoctrineExtensions#750 (comment)
• https://twitter.com/webdevilopers/status/720289162142744576

A possible workaround could be a custom mapper:

• https://github.com/akeneo/pim-community-dev/blob/master/src/Akeneo/Bundle/StorageUtilsBundle/Doctrine/MongoDBODM/Types/Entities.php by @nidup

A different solution could be treating referenced entities as Value Objects which at the same time could serve as a historical "query database".

For instance:
Instead of linking an ODM Order Document (still an "Entity" in DDD though) with an ORM Customer you introduce a Customer Value Object that keeps Id and full name. Maybe the delivery and / or invoice address too.

Remember kids:

Implementation issues (such as persistence) are not dealt with in the model, but they must be in the design.” - @ericevans0
https://twitter.com/fromddd/status/725604978564308992

[webdevilopers]

Regarding your mapping suggestion @nidup:

• https://github.com/akeneo/pim-community-dev/blob/master/src/Akeneo/Bundle/StorageUtilsBundle/Doctrine/MongoDBODM/Types/Entities.php

I'm trying the other way around:
Order Document has one Customer Entity.

So I need an extra "Document(s)" Mapping type, right? Do you have a (XML?) mapping example from your use case?

[nidup]

Hi @webdevilopers

Thank you for opening this discussion and for sharing these links!

Pretty sure that @jjanvier will be very interested by this issue too, he very well know the topic and he's preparing a talk about our hybrid storage for a sfpot this week http://www.meetup.com/fr-FR/afsy-sfpot/events/230197553/ ;)

To give a bit more details, we use a hybrid storage in the context of Akeneo Product Information Management system (https://www.akeneo.com/).

Depending on the amount of product data, we can use,

• a full Doctrine ORM storage with EAV implementation for product data
• a hybrid storage with Doctrine ORM for standard entities (family, category, attribute) + MongoDB ODM for the product data

When we implemented the hybrid MongoDB storage, the difficulty was to,

• keep the same business code for full ORM and for hybrid storage ORM + ODM (we use several interfaces for querying and persistence)
• make "transparent" the link between ORM entities and ODM documents for the dev who's working on features

To do so we introduced a StorageUtilsBundle and we configure the storage to be used like this https://github.com/akeneo/pim-community-dev/blob/master/app/config/pim_parameters.yml#L35

Let's take the product MongoDB mapping https://github.com/akeneo/pim-community-dev/blob/master/src/Pim/Bundle/CatalogBundle/Resources/config/model/doctrine/Product.mongodb.yml

This product document is linked to a single family (managed by Doctrine ORM),

        family:
            type: entity
            targetEntity: Pim\Component\Catalog\Model\FamilyInterface

It's also linked to several categories (managed by Doctrine ORM),

        categories:
            notSaved: true
            type: entities
            targetEntity: Pim\Component\Catalog\Model\CategoryInterface
            idsField: categoryIds

These new entity and entities MongoDB types are defined here https://github.com/akeneo/pim-community-dev/tree/master/src/Akeneo/Bundle/StorageUtilsBundle/Doctrine/MongoDBODM/Types

We store entity ids in the MongoDB document and we use MongoDB ODM events to convert these ids into lazy entities,
https://github.com/akeneo/pim-community-dev/tree/master/src/Akeneo/Bundle/StorageUtilsBundle/EventSubscriber/MongoDBODM

To build collection of lazy loaded referenced entities, we use https://github.com/akeneo/pim-community-dev/blob/master/src/Akeneo/Bundle/StorageUtilsBundle/Doctrine/MongoDBODM/Collections/ReferencedCollection.php

We also use the following listener to be able to use an interface resolved as a Doctrine ORM entity in a Doctrine MongoDB ODM mapping (for instance Pim\Component\Catalog\Model\FamilyInterface) https://github.com/akeneo/pim-community-dev/blob/master/src/Akeneo/Bundle/StorageUtilsBundle/EventListener/MongoDBODM/ResolveTargetEntityListener.php

The StorageUtilsBundle is already open source and not coupled to our PIM business code but unfortunately, for now, it's part of our main dev repository.

If after your first tests, you're interested to use it, don't hesitate to ping me, we can extract this bundle in a dedicated repository with a subtree split and register it on packagist to make it easily usable in others projects.

[webdevilopers]

Thanks @nidup for this detailled answer!

Looking at your mapping example:
https://github.com/akeneo/pim-community-dev/blob/master/src/Pim/Bundle/CatalogBundle/Resources/config/model/doctrine/Product.mongodb.yml#L58-64

So you still have an "extra Field / Collection" to keep the IDs that have to be converted in some way?

This is what I was concerned about from a Domain-Driven Design POV. Isn't the Infrastructure / Persistence having a little bit too much impact on the design of the DomainModel?

Coming from twitter discussion w/ @carlosbuenosvinos @eneko here:
https://twitter.com/webdevilopers/status/720289162142744576

[webdevilopers]

My current workaournd looks like this:

I store my Customer via ORM (MySQL). Orders are stored in ODM (MongoDB).
I have an customer property on my ODM Document. There is no extra customerId.
The customer property itself is an embedOne reference. It holds the historical data - I guess you could say "Aggregate" - typically:

• ID
• name
• billingAddress

The namespaces look like this:

• Acme\QualityManagement\DomainModel\Customer\Customer.php (CustomerId, CustomerRepository etc.)
• Acme\QualityManagement\DomainModel\Order\Order.php (OrderId, OrderRepository etc.)
• Acme\QualityManagement\DomainModel\Order\OrderCustomer.php
• Acme\QualityManagement\Infrastructure\Persistence\Doctrine\DoctrineCustomerRepository.php
• Acme\QualityManagement\Infrastructure\Persistence\Doctrine\MongoOrderRepository.php

I think the idea is similar to the "Aggregates and Event Sourcing A+ES" chapter of the red book by @VaughnVernon . With ORM an Event could update a Query (READ) Model that writes the historical data / Aggregate into a database VIEW.

Using embed references in ODM is great to take a final snapshot.

If I ever need more data from the Customer I would inject the CustomerRepository into my Query (READ) Model and get it from the ORM.

By storing the ID inside the reference I can still query for "OrdersByCustomer".

Actually I do have an extra column too but it does not feel like polluting the DomainModel for a persistence purpose.

Though I will still have to use custom mappers or types like @nidup did to convert those IDs into Entities when calling collections on my DomainModel e.g.:

class Customer
{
    private $orders; // some EXTRA-LAZY Doctrine ArrayCollection

    public function getOrder(OrderId $orderId)
    {
        return $this->orders[$orderId];
    }
}

What do you think of this approach?

[jjanvier]

Hello @webdevilopers

Maybe this presentation will interest you. https://speakerdeck.com/jjanvier/products-storage-at-akeneo-pim

It describes how we store our products at Akeneo PIM. At the end of the presentation, you have the most interesting part of the code that are used to link ORM entities to MongoDB documents.

[webdevilopers]

Thanks @jjanvier !

At the bottom line you are using an approach to the DoctrineExtensions by @Atlantic18:
https://github.com/Atlantic18/DoctrineExtensions/blob/master/doc/references.md

But you also support converting collections to Entities?

Still you have an extra Field e.g. categoryIds allthough you have categories already.
Not on the mapping only but also on your Document - the DomainModel - right?

This is what I am mainly concerned about from a DDD POV and that's why I asked @carlosbuenosvinos what he thinks about this approach. Maybe the other contributors @eddmann, @keyvanakbary and @theUniC have an opinion on this.

Technically I agree with your approach and would handle it the same way!

[webdevilopers]

One final way to solve this and keep the DomainModel "clean" is to create a DoctrineCustomer Entity or MongoOrder Document that adds the extra field.

[webdevilopers]

Currently looking at a CQRS post by @martinfowler:
http://martinfowler.com/bliki/CQRS.html

He states:

Developers typically build their own conceptual model which they use to manipulate the core elements of the model. If you're using a Domain Model, then this is usually the conceptual representation of the domain.
You typically also make the persistent storage as close to the conceptual model as you can.

If I regard my Domain Model being a representation I could build my Order Document this way instead:

class Order
{
    private $customerId; // Used to get the related Entity - just the ID or the CustomerId Value Object?
    private $customer; // The related Entity
    private $customerName; // snapshot / aggregate / historical data - bad example since `name` will mostly stay the same :)
    private $billingAddress; // snapshot / aggregate / historical data
}

[nidup]

Hi @webdevilopers!

I'm very interested in the DDD but I don't know it enough for now, I'm actively documenting on this topic :) (so this answer may contain some misuses of the terminology or weird approach 😜 ).

So you still have an "extra Field / Collection" to keep the IDs that have to be converted in some way?
This is what I was concerned about from a Domain-Driven Design POV. Isn't the Infrastructure / Persistence having a little bit too much impact on the design of the DomainModel?

Absolutely, I do agree with you, the categoryIds field is a persistence concern and should not pollute the domain model. When we wrote the original implementation (in 2014, times fly!), we tried to find a very technical solution about how to "build" and "use" this relation between Product Document and Category Entity without taking too much care about the impact on the domain models.

One of the biggest difficulty we had was to build an efficient and abstract way to do complex queries on these objects from the business layer POV. This point also pushed us to pollute the Product model by adding a "normalizedData" field to help us to write MongoDB queries https://github.com/akeneo/pim-community-dev/blob/master/src/Pim/Bundle/CatalogBundle/Resources/config/model/doctrine/Product.mongodb.yml#L78 (practice quite common when using a MongoDB storage to store the data in a way which simplifies the usage).

In a perfect world, imvho, you should not even care of the persistence of your Domain models when defining and coding them because persistence is a pure technical implementation detail. In real life, it's not that easy to decouple these objects from Doctrine ORM, I'm not even sure that's a good idea to try to directly store these domain models as entity/document.

One final way to solve this and keep the DomainModel "clean" is to create a DoctrineCustomer Entity or MongoOrder Document that adds the extra field.

I think this approach is really really interesting, your persistence classes can rely on these Entity/Document for their very specific concerns (fields used to keep a relation between storage, reference collection, etc) keeping a clean DomainModel.

Your business code could define and rely on a "repository" interface allowing to fetch domain models.
The implementation of this interface in infrastructure could internally fetch Doctrine document / entities but return Domain models only (never expose entity/document and taking care to convert / hydrate).

I recently did experiments around Hexagonal Architecture (Ports & Adapters) on a reporting project and achieved to easily decouple Domain from Persistence using a quite similar strategy. A DatabaseAdapter taking care of implementing querying and converting data from storage "objects" into Domain models (I didn't used Doctrine ORM for this project, I used https://github.com/pomm-project in fact only the foundation part + custom code for converting objects).

Thanks again for opening this discussion, I'm looking forward to see other opinions / ideas.

[webdevilopers]

I'm as excited as you are @nidup .

BTW just found this quote here by @vkhorikov:
If you tend to write code like in the sample below, you are on the wrong path <- Persistence Ignorance

• http://enterprisecraftsmanship.com/2014/11/29/entity-framework-6-7-vs-nhibernate-4-ddd-perspective/

;)

Not sure if the approach of a Database related Wrapper like DoctrineCustomer or MongoOrder can be compared to SQL Speaking Objects introduced by @yegor256:
http://www.yegor256.com/2014/12/01/orm-offensive-anti-pattern.html#sql-speaking-objects

Wrapper seem to be an appraoch indeed:
https://twitter.com/electricmaxxx/status/725346621005922304 @dbu @dantleech

[webdevilopers]

BTW the extra IDs are mentioned in the DDD in PHP book too. There they are used as surrogateIds.
These are not used by the Domain. But they are required by Doctrine to be able to persist Value Objects.

The book also suggests to using a Layer SuperType:
http://martinfowler.com/eaaCatalog/layerSupertype.html

Maybe the "polluting extra id" could be moved to an abstract class e.g. "HybridDomainObject" which then can be extended by the Order Domain Model.

[webdevilopers]

Nice article here btw:
http://scabl.blogspot.de/2015/03/aeddd-2.html

Most Java/Spring/JPA projects have separate repository and service layers. Ostensibly, the persistence logic and persistence concerns are encapsulated in the repositories, often called data access objects (DAOs). But in every JPA project I’ve worked on, this encapsulation fails, and the service layer is intimately involved with persistence concerns. It has to be aware of fetching strategies, cascades, and special handling of proxy objects. And it has to manage flushing the persistence cache, and understand which entities need to be reloaded into the cache before continuing after a flush.

In the end, there are many situations where we end up having to make compromising decisions within our domain to get the ORM mapping to work right.

[vkhorikov]

Hi, thanks for the invitation, @webdevilopers

If I understood correctly from the conversation, you suggest creating sub-classes to keep domain classes clean, smth like this: ClassWithPersistenceConcerns extends CleanDomainClass.

IMO, that's a good solution. In fact, that is exactly how ORMs like Hibernate approach the problem: they create sub-classes (proxies) for your domain classes at runtime and use them to handle persistence logic.

The only concern I have about this solution is complexity. Implementing such proxies manually can become a hurdle and one should weight pros and cons of this approach really carefully. It might be that purity you will get out of it doesn't worth the amount of work it requires. But this is worth trying anyway, at least as a research project.

BTW, I agree with your opinion on Ids in domain classes. Here I wrote a post on that: http://enterprisecraftsmanship.com/2016/03/08/link-to-an-aggregate-reference-or-id/

[webdevilopers]

Welcome @vkhorikov !

Are there Events and Event Subscribers in (N)Hibernate too? Do you have any experiences with ORM and ODM (OGM) hybrids in Java or .NET?

I really liked your article on this topic. Good to know I'm not the only one concerned about it! :)

Here is my current PHP approach with Doctrine:

As I described before in my use case I have a Customer Entity which has Document Orders.
The Doctrine Cookbook uses the extra CustomerId we were talking about:
http://doctrine-orm.readthedocs.org/projects/doctrine-mongodb-odm/en/latest/cookbook/blending-orm-and-mongodb-odm.html#event-subscriber

In my example I kept the property on the Order Document named $customer. The expected name for the DomainModel. Instead of being a placeholder I set here the actual ID of the Customer e.g. 969 when persisting.

Now I changed the Event Subscriber from the Cookbook to "convert" / "transform" this field into a Document Collection instead of using the extra field:
https://gist.github.com/webdevilopers/e65f795749c40601276f60a5a3517a85

This works fine so far.

It's just a rapid prototype. I will heavily test it. But it looks promising in the first place.

[webdevilopers]

I've improved the ORM listener and added an ODM listener to get a referenced Entity Supplier which is stored as an Id inside the Order Document with the expected Domain Model property named $supplier.

So far everything seems to work fine without any extra ID.

What do you think? Any bad practice using Doctrine Events this way? Any Hibernate experts here? @thmuch and @brmeyer maybe? :)