Thursday, May 1, 2014

Asset-rack and Nodemon for Easy Caching in Node.js

Caching of static assets (JavaScript, CSS, images) is one way to increase the performance of your website. Caching decreases the amount of bandwidth used, and when done right, it also decreases the number of HTTP requests that the browser must make to load the page.

One way to cache assets is with ETags. When an asset is requested for the first time, the response includes the ETag header. On subsequent requests, the browser sends the ETag back to the server via the If-None-Match header. The server then compares the ETag it received from the browser to the current version of the ETag. If the ETag matches, it responds with a 304 Not Modified and an empty body which saves drastically on bandwidth.

The Cache-Control header allows the server to give more complex instructions to the browser about caching behavior. When max-age is specified, the server can tell the browser to keep that asset and not ask about it again for a specified number of seconds. For example, Cache-Control: max-age=3600 tells the browser that the response should be cached for 1 hour. That not only cuts down on bandwidth but also HTTP requests.

There is a danger is telling the browser to cache and not validate whether a resource has been updated recently. If you update the HTML structure of your page but your CSS is being cached for an hour, then your site could look broken until the browser checks to see whether the CSS has been updated.

Fingerprinting solves that problem. Instead of referring to mysite.css in the HTML, we calculate a fingerprint (md5 hash) of mysite.css and add it to the file name so that now we have mysite-29fe7a48516763c7c3033929e3d4c2a0.css. The fingerprint will change when the CSS changes which changes the filename. Now we know these files will never go stale and therefore can be cached forever. We can set max-age to one year which is the maximum value for max-age. Now when we update our HTML and CSS, the filename of the CSS changes so the browser asks for the new file.

You don't want to calculate these fingerprints on your own and update your HTML by hand to refer to the new versions. You want a framework or build process which does that automagically for you. We use node so we found asset-rack. In our HTML templates we just call assets.url('/mysite.css') and it handles all the magic of fingerprinting for us.

There was a little heartburn during development at first because we had to restart the app everytime our CSS changed so that asset-rack could update it's own cache (ironically). Nodemon solved that problem for us. We configure it to watch all CSS and JS files and automatically restart the app when anything changes: nodemon -e js,css app.js.

Now we don't have to think about caching.

Further reading:

Friday, April 18, 2014

Find-A-Record is Growing Up

Find-A-Record has grown to the point that it needs it's own blog so we set one up at This also means we will cease to regularly post on this blog. It's possible that we will continue to post updates about Open Place Database, or share development war stories that don't belong on the Find-A-Record blog, but they won't be regular.

We enjoyed blogging for the past 7 months, and appreciate those of you who followed us most of the way. A special thanks to the die-hard fans that watched the videos. We hope you will continue to follow the progress of Find-A-Record. We expect some great things to happen.

Wednesday, April 9, 2014

Watercooler Wednesday #32 - Free, Online, and Military Search Filters

We added the ability to filter Find-A-Record search results based on whether they are free, paid, online, or offline. The only free collections we have right now are from FamilySearch: the free and online collections are the digital collections at FamilySearch while the free and offline collections are from the FamilySearch Catalog. All other collections we currently have are paid and online.

You'll see that we also added the "Military" record type for searching. We have military records from Fold3 and FamilySearch. We will add more military records from Ancestry, the FamilySearch Catalog, and other places in the future.

Our Repositories page contains the full list of sources for our collections, as well as some stats about our coverage.

Wednesday, April 2, 2014

Watercooler Wednesday #31 - WorldVitalRecords

Today we added collections from WorldVitalRecords. We were able to import about 20% of the 22 thousand collections that WorldVitalRecords has. That percentage will go up overtime as we continue get better insight into our data and indexing process.

We also increased the number of Ancestry collections from 7% to 26%. You can see the details on our repositories page.

Stay tuned for many more updates on new collections; we expect to add many more in the next few weeks.

Wednesday, March 26, 2014

Watercooler Wednesday #30 - Collection Details View

Yesterday we released the new collection details view. Each collection now has its own page that allows you to take a peek into how we indexed the collection.

Above is a screen shot from the collection page for the United States Marriage, 1733-1990 index from FamilySearch. The orange area represents it's coverage between 1803 and 1812. If you search for marriages in Florida in 1805, this collection will not show up.

These new collection pages are beautiful, fun, and useful. We have already identified and fixed several bugs in our indexing process now that we can easily visualize our data. It's also useful for researchers to understand the coverage of particular collections. For example, when searching for records in Eastern New Mexico, it's easy to see why collections from Texas appear in the results.

These new collection detail pages are available from the search results. Just click the "Details" button to open that collection's detail page in a new tab.