- When saving really high resolution polygons (such as Alaska), we discovered that CloudFlare's max request entity size is 100 MB.
- When creating a snapshot zip file for OPD, and later when process that data to load it into Find-A-Record, we found that node.js has a max buffer size of 1GB. We had to rewrite two scripts because of that.
- Simplifying countries with a lot of islands and crazy coastlines often results in invalid shapes. We ended up using MapShaper for simplification and repairing. Sadly, it doesn't have a documented node API so we had to learn that too by reading source code.
- We really are liking the nice weather we've been having. 60+ degrees equals running outside :)
- Trying to import 1 million collections from the Family History Library Catalog takes a wee bit o' time.
- Simplifying and saving hundreds of thousands of shapes takes a long time.
- When making thousands of HTTP requests, chances are at least one of them is going to fail so you better at least have error handling if not also retries.
- We found out that Justin's keyboard hates him and that he needs a new one.
Wednesday, February 26, 2014
Watercooler Wednesday #26 - More data = new problems
While loading more data into Open Place Database and Find-A-Record, we have learned the following: