Goals & Criteria

A broadly approachable services layer technology lets us reach out to a large number of Web developers, allowing them to get involved in the Engage community and technology using skills and languages they may already be familiar with. Similarly, the Fluid community itself needs to hit the ground running with Engage, building the services layer in technologies that are familiar and will get us coding with minimum ramp-up time.

Interoperability is a primary goal of Engage, ensuring that our technologies will work with a wide range of existing authoring tools, content management systems, and other tools commonly found in museums and on the Web. The Engage services layer needs to fit in well and leverage open standards wherever possible.

At the same time, our choice for server-side technology needs to be scalable and forward-looking, allowing museums to invest in Engage over long run, without fear that our services will slow down impossibly when confronted with large collections, and that they won't quickly go obsolete.

Lastly, the Fluid community has built up, over the years, a set of techniques and philosophies for writing software: accessibility, the Open Web, functional programming, and markup agnosticism are common themes throughout our technologies. As a community, we embrace diversity and a wide range of styles; shared patterns and techniques in code help make our solutions more coherent and consistent to develop with.

Some Quick Background Information

We explored a variety of potential technologies for the Engage services layer. In each case, we considered both a programming language and an accompanying Web framework, recognizing that much of the advantage of a particular language comes from good tools we can reuse. We looked at solutions in Python, Ruby, PHP, and JavaScript running on the server.

Our aim was to find a language and accompanying framework that fit both the technological needs of our users and the culture of our development community. Throughout the process, we carefully examined both features and the wider context: the associated community, documentation, and support.

In-depth information about our approach to evaluating technologies, along with our observations, is available at the Fluid Engage Server-side Technology Notes page.

CouchDB and Lucene

Data Feeds and CouchDB

There is an incredible amount of diversity among museums in terms of how they each organize their collections and structure their data. Each museum's collection is different, and it is a daunting and risky task to attempt to force all types of collections into a single schema. Engage will need to embrace this diversity; rather than creating a "one size fits all" approach at the database level, we'll take a flexible approach to schemas in Engage.

Given that we can't pre-bake the data model completely, a standard SQL-based relational database in inappropriate for Engage. A new crop of document-oriented databases has emerged that are well-suited to handling dynamic schemas. Foremost among these is CouchDB, a highly scalable and Web-friendly database written in Erlang.

Couch specifically addresses the schema problem we face in Engage. Data is stored as documents in JSON format, making it particularly useful for use in client-side applications that make heavy use of JavaScript. CouchDB Views, the means for filtering, aggregating, and retrieving data from documents in the database, are written in JavaScript and can be used to make various data feeds from a collection of documents. This provides a solution that fits neatly within Engage's open Web architectural approach, and is approachable to a wide range of developers.

Data in Couch is queried and persisted using a RESTful API: fetching data requires a standard HTTP GET request, while adding or updating documents is just a PUT or POST operation. This means that Couch can be used with any programming language without required native database drivers. It's also an excellent fit for a RESTful service-oriented architecture. For museums that already have their own databases or CMS systems, Couch can be replaced with a custom service that provides the same RESTful operations and feeds as provides by the Couch Views that will ship out of the box with Engage.

Written in Erlang, Couch is extremely scalable and can meet the requirements of massive, complex collections and exhibit data.

Free Text Searching With Lucene

Users have come to expect natural, Google-like text searches from most modern Web experiences. As a result, Engage will need to include a free text search engine service.

Hands down, the most viable open source technology for this is Apache's Lucene. Today, it's the fastest and most reliable open source text searching software available, and it has few competitors. Lucene is a JVM-based tool, and should be relatively easy to install in most server environments. Lucene can be easily connected to CouchDB, allowing new documents in Couch to seamlessly be added to the indexes.

We will continue to assess Lucene's associated Solr project, which provides a RESTful API and JSON-based wire protocol ideal for Engage's architecture.

JavaScript on the Server

JavaScript is traditionally considered as a client-only language, running mostly in Web browsers. A number of runtimes do exist that enable JavaScript as a general-purpose programming language, and most modern browser engines can be compiled as standard libraries for use on the server.

At the same time, JavaScript is probably the most widely used and understood programming language today. Regardless of what technologies they use on the server-side, the majority of Web developers also write client-side code in JavaScript. In terms of the ubiquity angle, JavaScript is overwhelming familiar and accessible to programmers coming from diverse background and technology camps.

That said, Web development on the server with JavaScript is still largely uncharted territory. There are few production-grade environments available for writing JavaScript Web applications on the server. A multitude of lesser-known projects exist, but few have picked up the momentum and community of more popular environments such as Ruby or Python.

As more richer and more complex application code is built in JavaScript on the client side, there is an increasing need to share code and development tools between the client and server. The Fluid community has built up an extremely productive toolset for developing usable and accessible interfaces with Infusion, our JavaScript application framework. As we extend our development efforts to Web services and server-side applications, a goal is to be able reuse as much of the infrastructure we've already got, lowering the cost and improving our productivity.

JavaScript on the server, while still on the bleeding edge, provides us with an broadly approachable environment for writing server-side code using many of the existing tools offered by Fluid Infusion.

Portability and JSGI

Next Generation Runtimes

Rhino and the JVM

Tying it all Together