Wednesday, February 26, 2014

Watercooler Wednesday #26 - More data = new problems

While loading more data into Open Place Database and Find-A-Record, we have learned the following:

  • When saving really high resolution polygons (such as Alaska), we discovered that CloudFlare's max request entity size is 100 MB.
  • When creating a snapshot zip file for OPD, and later when process that data to load it into Find-A-Record, we found that node.js has a max buffer size of 1GB. We had to rewrite two scripts because of that.
  • Simplifying countries with a lot of islands and crazy coastlines often results in invalid shapes. We ended up using MapShaper for simplification and repairing. Sadly, it doesn't have a documented node API so we had to learn that too by reading source code.
  • We really are liking the nice weather we've been having. 60+ degrees equals running outside :)
  • Trying to import 1 million collections from the Family History Library Catalog takes a wee bit o' time.
  • Simplifying and saving hundreds of thousands of shapes takes a long time.
  • When making thousands of HTTP requests, chances are at least one of them is going to fail so you better at least have error handling if not also retries.
  • We found out that Justin's keyboard hates him and that he needs a new one.

No comments:

Post a Comment