10
July 2009

Object Relational Database – The Node Mesh

Written by Tim Blair White
avatar

In confronting the complete redesign of our flagship website – Govtech.com – it was decided to embark on developing a proprietary object relational database that is affectionately known as The Node Mesh.  A full API and CMS were developed on top of the Node Mesh and Govtech.com has been running on this platform for the last two years.

The Node Mesh has provided a very flexible infrastructure for powering the company’s websites and has cut down development time by as much as 50% in some cases.  It is written in PHP5.

Understanding the Node Mesh

I will warn at the outset that the node mesh is non-traditional.  It’s unlike typical data structures you may be used to working with.  The approach of the node mesh is so radically different from traditional design that developers who start working with it without changing their paradigm on data structures spend way too much time barking up the wrong tree when troubleshooting.  Here is a list of common pitfalls that developers make if they don’t understand the node mesh:

  • Asking what table the data is stored in.
  • Searching for non-existent FKs.
  • Opening up enterprise manager too much.

The idea here is that to work with the node mesh you need to abandon notions of traditional design.  Traditional design works against many node mesh principals.  So brace yourself, open your mind and enjoy your journey to understand the node mesh.

Fundamental Building Blocks

We refer to chunks of data in the mesh as nodes.  Nodes have properties and are linked with other nodes, they are the first fundamental building block of the node mesh.  In essence everything is a node (even properties of a node can be broken out into nodes, more on this later).  Nodes also have types.  A node’s properties are defined by its type.  So for example an employee node has name, phone and email properties among others while a vehicle node has make and model properties.  Both employees and vehicles are node types so all employee nodes and all vehicle nodes have the same properties respectively.

The second fundamental building block of the node mesh is the link.  Links hold nodes together to create logical data structures.  All links are equal (or as we sometimes say all nodes are equal) in the mesh.  That means there is no inherent hierarchy or other special relationship, you simply link two nodes together, neither one is a parent or a child, ever.  Think of it as the Internet.  Hyperlinks contain no inherent special relationships, just a connection to the other location.  Amazingly everything can be accomplished with dumb links (links that contain no data) including the appearance of hierarchy and other special relationships.  This is handled by creating specific node structures tailored to the task.

That’s it.  Two building blocks for the node mesh: nodes and links.  Working in a system this simple makes development much easier.  So lets back away from the trees and take a look at the forest.

The Node Mesh Paradigm

Shape

Lets paint the picture of the mesh as a sphere.  Your life will be much easier if you look at everything in the mesh as a circle rather than a square.  The node mesh itself is a giant sphere and all the data within it is also spherical.  This shape carries over into node mesh diagrams which are mainly comprised of circles and lines. Consider the contrast between these two images below:

Nodemesh1

Here we get a clear visual of how the data operates.  Thinking of data in these terms is important because it changes the way you approach problems and solutions.  For example, when looking at the data in the image on the left you would ask: “which one of these nodes am I looking for and which path do I take to find it?”.  If you are looking at the image on the right you would ask: “which one of these data blocks am I looking for and which level, sector or row is it stored at?”.  Just from the shape of data alone the approach to the problem changes significantly.

Nature

The node mesh is also better understood when compared to nature.  For example molecular structures forming different combinations of matter present a pattern implemented in the node mesh.  As different molecules link together in nature a complete system is formed.  In the node mesh when different nodes link together they form a system as well.

If we back out further we can look at the brain as a system.  It is comprised of neurons and neurotransmitters that send signals between the neurons.  These pathways link the neurons together and allow the brain to process its data.  Similarly the node mesh is comprised of links between nodes that create pathways.  The pathways are used to retrieve and process data.

We can back out even further still and look at the body as a whole and how all its systems link together or even look at life generally and how everything is connected at some level.  This rather philosophical approach is an important part of the node mesh paradigm because understanding the node mesh is all about understanding connections and that ultimately anything can connect to anything else in the node mesh.

Notice that in nature data is organic, flexible, living and breathing.  Contrast this with the traditional data models of fabricated circuitry which are very rigid, fixed and “dead”.  The following images illustrate this point.

Nodemesh2

The above image represents the actual table structure used in an events system for the company.  There are 28 tables and 30 relationships between them.  Each relationship is custom containing special keys in some of the tables and requires maintenance.  This structure is designed to accomplish a specific task, it is far from flexible and requested changes are costly.  Now lets have a look at the same system in node mesh format.

Nodemesh3

This image illustrates a few principles.  First, both systems accomplish the same task but the node mesh version uses considerably less data.  Instead of 28 tables we have only 14 and in lieu of 30 relationships we have 15.  By following natural patterns we have reduced the footprint of the data structure by 50%.

But this is just the beginning.  In fact the smaller node mesh structure accomplishes more operations than its traditional counterpart.  In addition to that the relationships between the nodes are standardized meaning other nodes can be introduced into the system with ease.  In the traditional model a complicated network of primary and foreign keys varying with each table must be accounted for.  Not so with the node mesh.

Finally, the diagrams speak for themselves.  The first is rigid, overbearing and completely manufactured.  Because of the way that relationships are handled table columns are necessarily shown.  The second is simpler, feels natural and is just plain sexier.  Additionally there is no need to show the properties of the node unless you want to as relationships are not tied to those properties.

Scale

One final and very important part of the node mesh paradigm is understanding scale.  For traditional models scalability is frequently not an issue if all you are scaling is the data.  But if you want to scale the model itself you are out of luck as those types of expansions are costly.  Contrast that with the node mesh design.  Its design pattern is scalable by very definition.  The node mesh deals with only two things, nodes and links.  That allows you to move in both directions on the scale very easily.

For example, currently we implement most nodes as data objects with properties.  However it is arguable that the properties of those nodes can be broken off as nodes themselves.  This is an example of scaling down.  Scaling up is just as easy, it involves the use of adding completely new nodes into the system, adding completely new systems and even adding new meshes.  Because all nodes can link to other nodes the possibilities of system structures are unlimited.

We currently utilize multiple meshes and term this behavior “multi-meshing”.  Multi-meshing is conveniently built into the API allowing you to setup another independent mesh and then later connect it with another mesh.  An example diagram below illustrates some hypothetical uses:

Nodemesh4

Here nodes from each mesh link up with nodes of other meshes.  This type of scalability allows you to create unlimited meshes linked to  each other.

Diagramming

As you have probably already gathered the node mesh isn’t going to fit into a standard ERD.  Thankfully so.  We created a diagramming model specifically for the node mesh.  This model has a number of details and standards but for now just understand the following basics:

  1. Nodes are represented as circles with the type in the middle in singular form (i.e. employee, vehicle).
  2. Node Clusters are represented as squares with the type in the middle in plural form (i.e. employees, vehicles).
  3. Solid lines indicate that nodes or clusters are linked.

These three rules will get you by when working on the white board and conversing with other developers.

API Examples

Finally, what would an overview be like without a little taste of the good stuff?

In the illustration of the node mesh under the shape paradigm we saw a series of nodes connected with links and asked the question: “which one of these nodes am I looking for and which path do I take to find it?”  The API provides the answer to this question.  If for example you want to find all the orders assigned to you the following query would do it assuming you are the $employee:

$myorders = $employee->jobs->orders;

$myorders = $employee->jobs->orders;

The node diagram looks like this:

Nodemesh5

Yes, it really is just that easy.  Welcome to the node mesh.

Leave a Reply