By Alexandra Stuckless on December 22, 2015
Lately, there’s been a giant demand for data and statistics that can be used to help make business decisions. If you’re working as a developer or engineer on a team that’s tasked with gathering this data, you’ve probably noticed that this demand has translated into more information and events being tracked, and more pressure on the systems you use to extract, transfer, and load your data. You may have even run into a situation where your current systems start to buckle under the increased load, and reliability and stability begin to degrade.
This year, the Datalab Team at Hootsuite ran into this problem and came to the conclusion that using Logstash in our data pipeline wasn’t working – we needed to migrate to something else. This blog post outlines what we did to make our transition smoother, and will hopefully provide some useful tips if your team is looking to change part of your extract-transform-load (ETL) system.