Software complexity is killing software development as we know it. I’ve held this opinion for at least the last decade, and I thought it would make a good first post for the Cyanic Automation developer blog.

If we look at how most major development environments evolve over time, it is clear that each new version brings increased complexity due to new language features, new frameworks and subsystems and new tooling. The march is endless and systems such as Microsoft .NET are continuing to grow new arms and legs every day. It’s always good to keep moving forward, but new features usually follow the same pattern: nice, user-friendly developer tools that promise simplicity, yet eventually suck you into a world of limitations, hacks, and a huge burden of knowledge as soon as you try to do something useful with them.

I hate to pick on Microsoft, but let’s have a look at what they’ve given us in recent memory (some of this may not be entirely fair, but I’m trying to prove a point here so please bear with me):

  • Entity Framework: Yet another ORM framework.  Many developers, encouraged by early successes of drag & drop database integration continue down this path and eventually get mired in compromised database structure, leaky abstractions in their data layer, bad query performance and RDBMS lock-in.  I’m not saying EF (or ORM in general for that matter) is bad, but these systems are huge and require a large time commitment to master.  I think we can at least all agree that the effort that has gone into engineering and using these systems just so that we can persist and query data with a bit less code, is pretty insane.  A great article on the topic that has stuck with me over the years: The Vietnam of Computer Science.
  • WCF Data Services: Gives you the ability to create an OData service from any EDM model with just a few mouse clicks!  Unfortunately, exposing your raw relational data probably looks like crap in OData, and if you want to customize the rendering or (god forbid) try to use a non-relational source you’re in for a world of hurt, most likely ending in you writing a custom provider, an IQueryable implementation and bottles of absinthe to dull the pain.  I once spent months of my prime programming years making a custom Data Services provider for a semantic data store, and memories of the experience still fill me with anxiety when my mind wanders to the topic.  The documentation for WCF Data Services is poor to non-existent, the only useful resources being MSDN developer posts that usually have to explore into internal APIs and end with a “happy hacking” directive.
  • ASP.NET Web Forms: Okay, this is going back a ways, but lots of people still use Web Forms, and it’s horrible enough to make the list.  Again, lured by the promise of building dynamic web pages by drag & drop, all Web Forms projects ultimately decay into something sick and twisted and terrible to behold.  One of the worst experiences a developer can have is to come into a Web Forms project cold, expecting to maintain it.  None of the data flow make sense, especially with 3rd party controls that magically communicate back to the server, with crazy eventing models and state management.  MVC usually fares much better in this regard, but is still a large platform to learn and master.
  • Windows WF: Now here’s a way to implement processes through drag & drop (instead of umm, writing code I suppose).  Unfortunately there’s a large burden of knowledge before you can do anything remotely useful with it, and you end up writing a whole lot of code to save yourself from writing small amounts of code.  Maybe I just don’t “get it”, but I’m not sure why developers are constantly discouraged from writing code – it’s what we’ve spent our lives doing, and we’re better at it than anything else.  Windows WF actually makes a lot of sense in SharePoint and it vastly makes that platform more capable; unfortunately, WF has been getting less useful in SharePoint over time, presumably as Microsoft takes away power features so it better fits into Microsoft’s low-trust hosting environments.
  • SharePoint: Here we are.  The mother of all supremely complicated, mis-engineered and confounding software packages.  This usually comes as a surprise to people; after all, SharePoint is becoming ubiquitous, and it’s really quite nice for the end user.  But, you really have to stand back in awe of how quickly it can grind down and crush the hopes of even the most hardboiled of developers.  As soon as you get past the user-friendly façade and try to write code to do anything, even the simplest customization becomes a living nightmare.  Nothing ever works the way you expect it, the APIs are inconsistent and bizarre in places, the documentation is either missing or wrong, none of the error messages or logs are helpful and some features are just plain unreliable (I’m looking at you, workflow!).  There’s so much time spent trying to get even basic things working, you start to wonder if you’re losing your edge – this is an enterprise-level Microsoft product, it must be me, right?  But then you realize, after all the research and Stack Overflow articles from equally suicidal developers around the world, that the whole thing is just completely rotten on the inside, going all the way back to its Vermeer/FrontPage origins.  At some point I’ll have to gather my thoughts and write a whole article about SharePoint, mostly to get it off my chest, but also to serve as a warning to others.  I’ve been developing with SharePoint since the WSS 3.0 days – I’ve eventually learned how to make SharePoint projects succeed, partly through a hard-earned repertoire of magic tricks and secret sauce, but mostly from knowledge about what parts of SharePoint are trouble and staying the hell away from those.

 

Why should you care about any of this?  For a few reasons:

  • The energy level of a software developer can be an amazing, but fickle thing.  A problem that is tough, but adds value when solved will excite and challenge a developer.  He or she will throw their whole being into solving a problem like that, and the end result is productivity and satisfaction.  On the other hand, when working on a problem that is simple but made difficult by the technology being used, a developer’s energy is ground down to nothing, producing mostly waste and frustration.  Morale on the development team is extremely important, and in my experience, technology choices play a role in keeping people happy.
  • Complexity tends to beget more complexity, and can affect entire cultures.  You can see this all through Microsoft, and even worse in the Java community, where people build frameworks for building frameworks, seemingly to feed some kind of masochistic deity that gains strength from the suffering of developers.
  • In today’s big-corporate environment, there just isn’t time to learn and master these large subsystems.  In truth, very little of a developer’s time is spent making things anymore as the burden of processes increases.  Here I’m talking about requirements traceability, quality metrics, stage gate completion checklists, the list goes on and on.  What happens when it’s time to write some code?  Frequently people will stick with ancient or inappropriate technology that they are comfortable with.  Or, they charge ahead with new technology they don’t know much about, but were lured into false security from the few quick tests that worked out well.
  • People tend to become emotionally invested or attached to things that took a lot of time and effort to master, and they become unwilling to make changes even when it makes sense.  There are few industries where a person’s working knowledge is obsoleted so frequently.  It happens all the time in the software development world, especially with companies like Microsoft who tend to lose interest and abandon entire systems (Silverlight, LINQ to SQL, etc).

 

Simplicity and Node.js:

Cyanic Business Automation Studio (the platform that powers our Cyanic HSE product) was initially born out of unwillingness to accept a few key limitations in SharePoint that made it intractable for our needs.  When we started development, the pain of my last big-corporate project was still fresh.  I was working on an alarm viewer application for mobile heavy equipment, a project that was a perfect intersection of difficult legacy frameworks and technologies from years past.  While it all turned out well in the end, it took way longer than it should have, and was overall a mind-shattering, frustrating experience for something so conceptually simple.  I vowed that the next opportunity we had to start a project from scratch, I’d throw away all of my least favorite technologies and start over with something that fits with our principles.  After I used Node.js for a few days, I knew it would be a perfect fit for Cyanic Business Automation Studio.

What is Node.js?  Essentially it is Google’s V8 Javascript engine wrapped up with an asynchronous eventing library, and a helpful and active ecosystem with libraries to do anything you could possibly imagine.  I don’t know very much about its origins or the motivations of the creators, but to me it feels like a wholesale rejection of the idea that software needs to be difficult and complicated.  It’s a system stripped bare, a return to first principles:

  • Here’s a system that you can pretty much learn in a single day, and uses a language you probably already use of a daily basis.  It removes all need to worry about thread concurrency, and yet still achieves huge performance for non-compute server applications.  Its packaging system npm is excellent, and deployment is usually very straightforward.  Oh, and it’s truly cross-platform, and totally free!
  • You get to use the same core language on both client and server.  In addition to being easier on the brain when switching between client and server, you get to share code between them as well.
  • If complexity begets complexity, then the inverse also holds.  Node.js’ first class web frameworks (namely, Express and RESTify) use a single pattern for all request middleware and routing functions, and it gives you huge control over all parts of the web application.  ASP.NET MVC after all of its iterations and OWIN modules still doesn’t come close.  It’s shocking to think that after a few minutes of looking through examples that you can be fully productive – probably 90% of my current Express/Jade knowledge was gained in the first few hours of using it.  Further to this point most of the community contributed libraries are equally simple, a quick look through the standard project ‘readme’ and a few examples are enough to get you going without trouble.

 

How to get started with Node.js:

  • The first step is to buy and carefully read JavaScript: The Good Parts by Douglas Crockford.  This is the Kernighan & Ritchie book for Javascript, and tells you everything you need to know with a minimum of fluff.  If you already know Javascript (like I thought I did), this book does a great job at purging all of your harmful Javascript habits and preconceptions, and getting you to understand its true nature.  Javascript gets a lot of hate from certain developers (such as me, from about a year ago) but I now see it as a beautiful & massively expressive language that is a pleasure to use.
  • Download Node.js and your preferred development environment.  I personally use Node.js Tools for Visual Studio as it gives a similar experience to .NET development, including Intellisense and debugging.  I’m not sure if I’d recommend this at the moment, as the plugin is still pre-release, and still seems to have a minor case of serious brain damage.  I do have high hopes for this project however, and as soon as the bugs are worked out it’s going to be awesome.  Some other options to consider in the meantime:
    • Nodeclipse: I’ve tried it and it seems ok, though in general I much prefer Visual Studio to Eclipse.
    • A good Javascript editor such as Brackets or Atom, and debugging in Chrome through node-inspector.
  • Get familiar with Express, Jade and these high-quality libraries:
    • async: The only module that I consider to be mandatory.  If there’s one problem people complain about with Node.js is that nearly all calls are asynchronous (a good thing), but nested asynchronous callbacks get weird and hairy pretty fast.  Async solves this problem, in its entirety, and has a very positive effect on the readability & maintainability of your code.  You must use it.
    • Moment.js and Moment Timezone: Fills in all of the missing gaps in Javascript’s time handling functions.
    • Winston: A good logging library.

 

While we’re on the topic, a few notes about the other pieces that round out our current technology stack:

  • Postgresql: No journey to simplicity is complete without a migration from SQL Server to Postgresql.  The internet has already said enough about how excellent Postgresql really is, but I’d also like to add that it’s really simple to install, configure and work with.  Installation is usually a one-line command on most Linux distros.  Configuration is typically adjusting 1 or 2 configuration files, even getting write-ahead log archiving to work is really easy.  Everything just works the way you expect, and it also includes support for its awesome json column type.
  • Knockout.js: A client-side MVVM framework.  There are a lot of MVVM frameworks out there, but Knockout.js is really easy to get started with and is as powerful as you need it to be.  Perhaps most important is that it’s unopinionated software and integrates easily into the way I like to do things.
  • jQuery mobile: I’m less enthusiastic about this one, but it more or less works and has not entirely unacceptable performance.  At some point we’ll probably be looking to switch to another web framework.  (Have you had good experiences with another framework?  If so, please leave a comment below, we’re interested in your opinions).

 

Is Node.js all flowers, rainbows and unicorns?  Sadly, not entirely:

  • It’s still Javascript, and Javascript isn’t perfect.  It’s not a panacea, and no doubt other systems will emerge that are better in every way, but it’s a great choice for certain types of projects for today.
  • I’m not sure how well things will hold up for a large codebase and a large development team.  I suspect each project will need a strong-willed and loud-spoken enforcer to review code and make sure people are following conventions and prevent things from getting weird.
  • You do lose compile-time error checking, and some Intellisense fidelity.  That’s quite unfortunate, but I haven’t found it to be that much of a problem.  Plain syntax errors (missing bracket, broken object form, etc) are caught at start-up.  Invalid function calls and similar problems will sneak through, but are easy to fix and generally not the kind of errors I worry about.  In any case, it’s more important than ever to have reasonable automated test coverage.

 

In closing, I’d like to say that we’ve been very happy with Node.js in Cyanic Business Automation Studio.  It gives us the simplicity and power we need, offers great performance and stability, and is very comfortable with CBAS’s dynamic nature.  You should check it out, and see if it can make your job a bit more fun.

 

Happy hacking.