Learn about the technology, culture, and processes we use to build Hootsuite.

Everyone — our software developers, co-op students, high school students — 'works out loud' about the tools we use, the experiments we ran, and lessons we learned. We hope our stories and lessons help you.

Recent Posts:

At Hootsuite, we value innovation. We believe that it is important to allocate time to step back and tackle impactful issues. That is one of the reasons why we hold regular internal hackathons. Hackathons are a great way to get people thinking out of the box, sharing ideas, and growing their skills.

Hackathon 11/17 Logo

How does a Hackathon work exactly? First, employees pitch ideas (usually based upon a problem that they are trying to solve). Everyone has the option to either pitch online (through Workplace) or live on stage. Either way, they express their idea concisely and specify what skills and resources they are looking for to help bring the idea to life. Then everyone gets to decide what they want to work on. You can sign up with someone who pitched an idea or you can always do your own thing.

Ideas don’t have to be strictly technical, and non-technical people can always ask for technical help in their pitches, if they need it. We make hackathons inclusive because processes, media, and other aspects of the business can also be hacked.

Once everyone has a team the hacking begins! This time we allocated three days of hacking. On the last day everyone gets to demo what they achieved, good or bad. After all, you learn the most when failing.

Themes

This time around we had two main themes: “Enabling the Platform” and “Automagically”. The themes are there for focus, but also to encourage participants to align their hacks with company goals. You can always stray from the main themes and #gorouge as we like to call it.

The “Enabling the Platform” theme is geared towards the Hootsuite third-party app ecosystem. We want to have a great platform in which other developers can add-on capabilities. Hacks in this category would make it easier for developers to contribute to the ecosystem or improve the overall platform experience.

The “Automagically” theme looks to make our customers’ lives easier, or to automate their tasks. From batching of tasks to advanced artificial intelligence, anything that makes repetitive tasks easier or more intuitive qualifies.

The Pitches

We hosted the live pitches on the HQ1 main stage. The pitches were also broadcasted live for our owls in other offices to watch and participate. There were 15 live pitches and 24 online pitches for a total of 39 great ideas. The atmosphere at the pitches was very lively, a great way of getting everybody excited and ready for three days of hacking.

People then got a chance to find a team. Every person that pitched an idea was standing with the name of their idea on a piece of paper. Interested participants approached the idea holder and had a chat. From there, the teams were born. Form Birds of a feather to #hoggr, teams both large and small got to hacking.

People hacking People hacking People hacking People hacking

There were a total of 43 teams of which 13 hacked on the Automagically theme, 13 hacked on Enabling the Platform, and 17 were #goingrouge. The total number of participants was 129 people, almost all of our Product and Development teams across the world (some of them in more than one team #youmaketherules).

Demo Time

On Friday, starting at 1pm, everyone got a chance to demo their hack. Every team had three minutes to demo and one minute for questions. Our Toronto and New York offices kicked off the demos via live feed. Then we demoed Automagically-themed hacks in Vancouver followed by Enabling the Platform and finished off with #goingrouge.

People demoing

At the end of each theme group everyone got a chance to vote for the best hack in that group. This also allowed for much needed breaks.

And the winner is…

After four long but exciting hours of demos the votes were cast and we had the winners. We had 2 theme categories and 3 extra categories:

Automagically

Create AI Create.ai can predict the performance of a tweet as a user composes it. The team created and trained a deep neural network on a dataset of 50,000 tweets with the goal of predicting likes and retweets.

Enabling the Platform

APIs and webhooks The team, created a new integration for third-party developers. We don’t want to reveal too much, but keep an eye on the Hootsuite developers blog for a follow-up announcement.

Learning Opportunity (Failure)

Scoring Social Engagement Scoring user’s social behaviour from 0 to 5 using machine learning; social behaviour being your engagement with post and engagement on your posts. The idea is to help gauge a user’s social change after going through Hootsuite Academy training or using Amplify / Amplify + Selling.

Non-Technical Hack

Goodbyes are hard David rounded out his previous set of Hackathon animated emoji with a brand new set for slack.

Mobile Hack

Drag n’ Drop Apple introduced drag ‘n drop with iOS 11, allowing to easily move content from one app to another. Until now, the Hootsuite app was left out – no more! Drag ‘n drop allows to drag text, URLs, images and GIFs from any app straight to the Composer to attach them to the message. The feature will be available in the next release of the app.

Takeaways

A hackathon is a great way of getting a group of people to really focus on one thing. It’s amazing what a small team can achieve in a small amount of time.

Despite most of the participants being in Vancouver, it was a global effort with participants in Toronto, New York and Bucharest. So much so that a team from the New York office won best “Automagically” hack. As a global team which is working to be #bettertogether, it was great to see a project presented through the livestream get chosen as the best hack.

This hackathon produced a lot of great working implementations but it also was a great learning opportunity. Not all hacks were technical successes, but we recognize that failing is part of the learning process. Having a “Learning Opportunity” category in the prices encourages participants to stand in front of everyone and talk about what they tried and what they learned about it.

We will definitely continue to have hackathons at Hootsuite and we will improve how we do it to maximize impact and minimize friction. One thing we want to do better next time is the demos. Some demos are great but other ones can sometimes drag on for more than three minutes. We are thinking of ways to make this process better for next time.

Last week, I had the great honour to be an attendee and a speaker at the very first KotlinConf in San Francisco. As a self-described Kotlin evangelist, it was a great opportunity to be inspired by the stories and projects of other Kotlin developers from around the world. With three session tracks, and 1200 attendees, there was no shortage of things to do!

I wanted to write a post about my experiences at the conference while it is still fresh in my mind. All the sessions were recorded and will be uploaded to Youtube sometime this month, I will post again a link to my talk when they are public.

Speakers Dinner

In addition to the two days of conference proceedings, as a speaker, I was invited on a Hornblower cruise around the bay the night before the conference began.

I admit that I had to fight the feeling of imposter syndrome on the boat with some of the great names in Kotlin. I saw Andrey Breslav, the father of Kotlin, Dmitry Jemerov and Svetlana Isakova, authors of “Kotlin in Action” that we had previously read at Hootsuite in our Kotlin Book Club. I spoke with Hadi Hariri, myself and Simon Tse had previously appeared on his “Talking Kotlin” podcast. I also got a chance to thank Alina Dolgikh, who has been instrumental in the success of our Vancouver Kotlin Meetup Group (join at http://vankotl.in).

The cruise was great, we travelled out underneath the Golden Gate Bridge and braved the wind for a good view of the city. Sadly, my Pixel 2 has not yet arrived, otherwise my picture taken from above deck might have been clearer. Meeting some of the JetBrains team as well as the other speakers and connecting with them on a personal level was an experience that helped me to feel more comfortable speaking myself.

Conference Day 1

The conference began the following morning, in beautiful Pier 27, on the Embarcadero. Immediately upon entering the venue, we were greeted by all things Kotlin: banners, swag, even pillows on the couch!

Keynote

In the keynote, Andrey and team introduced some amazing new things for Kotlin. Kotlin Multiplatform Projects sound very interesting! In the future, you will be able to share code between your iOS, Android, and web clients as well as your backend servers. An official Kotlin wrapper for React is out now meaning you could soon swap jsx for kotlin, exciting!

There were a lot of great things announced so check out the KotlinConf Keynote Recap that JetBrains posted on the Kotlin blog.

The Morning Sessions

There are so many new stickers, I’m running out of laptop room!

Cords and Gumballs

The first session that I attended was Mike Hearn’s talk “Cords and Gumballs”. The first part of his talk was about how Corda is using Kotlin in the banking industry. Some of his stories of the way in which they trained devs on Kotlin in house reminded me of our own adoption here at Hootsuite. We found that it was very easy to have devs fall in love with Kotlin.

The second part of Mike’s talk was about a side project that he has been working on to package Kotlin into native go-like binaries. The tool, Gumball, is open-source, and he invites people to contribute to the project.

What’s New and Cool in Kotlin Tools

Dmitry Jemerov, the aforementioned coauthor of “Kotlin in Action” spoke on new features of Kotlin and its tooling. In this talk we got a closer look at Kotlin multiplatform. This initiative is incredibly exciting. Being able to share code across Mobile, Web Backend, and Web Frontend has the potential to save vast amounts of development time and effort. Additionally, in the new serialization library, protobuf is supported out of the box!

I had made it one of the objectives of my attendance at KotlinConf to get my copy of “Kotlin in Action” signed by both of its authors, after this Dmitry was kind enough to be the first.

Over lunch I ran into Svetlana and she was happy to sign as well, now I am the proud owner of a signed copy of the book.

The Afternoon Sessions

After getting some lunch, it was time for the afternoon sessions.

Kotlin Static Analysis with Android Lint

This talk with Tor Norbye was a real eye opener. The Android lint program is incredibly extensible and powerful. I can envision how much time we could save on pull requests if we wrote a few lint checks to automatically flag and fix common errors. My big takeaway from this talk is that our Android team needs to sit down for a few hours with Android lint.

Lessons Learned Building a Build Tool

The next talk I attended was Cedric Beust on the build system that he created, “Kobalt”. There were two things that stood out to me from this talk.

The first were the problems of figuring out what nodes of a dependency graph are safe to build. Simon Tse and I created a tool called “Peon” that automatically creates Pull Requests based on changes to dependencies in a graph. I will expand on the details of this tool in an upcoming blog post but the core similarity with Cedric’s problem was that we had to figure out how to create the minimum number of pull requests and in which order to merge them, to propagate changes through the graph.

The algorithm that Cedric used in Kobalt was a similar to (though more advanced than) the algorithm that Simon and I used in Peon. Cedric’s algorithm allowed for the parallelization of builds, it could be a great extension to the peon tool.

The other piece of the talk which stood out was Cedric’s discussion of Ad-hoc Polymorphism. His solution of how to implement it in Kotlin is similar (though more structured) than some of the things that we do in our own app to decouple the way we call APIs for displaying streams of social content. Both I and Andy Fisher, an Android developer at Hootsuite, who was with me at the talk, resolved to take this lesson back to our own code base, to perform ad-hoc polymorphism in a cleaner fashion.

Kotlin Puzzlers

The final talk of the day was Anton Keks speaking about “Kotlin Puzzlers”. His talk was inspired by Java Puzzlers and it had a very interactive format. Anton showed small snippets of code and had the audience guess the output of the program, then explain why they thought it was so.

The whole talk was great fun and full of head scratching and even counting exclamation marks for:

println(!!!!!foo!!!!)

In the end, I walked away from the talk with a smile on my face and a healthy distrust of the Kotlin compiler in certain circumstances. Check out Anton’s repo of puzzlers here: https://github.com/angryziber/kotlin-puzzlers.

Day 1 Evening

After the final session of the day, there was an evening full of events, a party in Pier 27 kicked off by a “Party Keynote”. I had to ask myself, “What the heck is a Party Keynote?”

Party Keynote

Thankfully our Party Keynote speaker, Michael Carducci answered that question for us. Starting with illusions on a software development theme and ending with some mentalism, including guessing that Stephanie Cuthbertson was thinking of her son Peter.

After the keynote, everyone was ready for a bite to eat. There was a lot of great food waiting for us, unfortunately everyone had the same idea and so it left to some long lines at the start.

After the lines settled down, there was still plenty for everyone and in the meantime, Andy and I were able to get a photo to commemorate the event.

Conference Day 2

After a great first day of the conference, I woke up on day 2 anxiously awaiting my presentation slot as the first after lunch. However, the first thing we had in store for the day was a talk from the always interesting, Erik Meijer.

My Life as a Tech Transfer Monad

I’m still reeling from this talk by Erik Meijer, on what exactly probabilistic programming is and how we as programmers can apply it to real world problems. However, There were two things I loved immediately from this talk. First, near the beginning, Erik had a very succinct description on the expressiveness of Kotlin.

The other is that, we as programmers need to learn to think and develop differently because machines are coming for our jobs vis-à-vis Machine Learning Algorithms.

My Talk

Now it was time for me to do some final preparation for my talk. I presented earlier this year at AnDevCon and I covered my experience here. KotlinConf was a lot larger though, with some 1200 attendees.

Kickstarting Kotlin Culture: The Journey from Java to Kotlin

I had been scheduled at the same time as Andrey Breslav and so I wasn’t sure if anyone would show up. However, I was pleasantly surprised when I had a room more than half full to present to.

As I mentioned in my talk, which I won’t go into detail until a later date when the video is published, I try to live by the principle of “working out loud”. Noel Pullen (who also wrote on Guilds at Hootsuite) gave me that advice on my first day at Hootsuite and it has really stuck with me. To me, working out loud means: Work passionately, know that what you’re doing will be valuable to someone, and strive to share what you’re working on with those around you.

This was a major motivator for me wanting to share our story of Kotlin adoption at Hootsuite. I hope that my talk helps others accelerate their adoption and to grow Kotlin culture in their own organizations and communities.

I was honoured to be able to tell our story at KotlinConf but I certainly had help doing so. I would like to thank my director, Paul Cowles, for supporting me and the team through our Kotlin journey here at Hootsuite, and for helping me tell this story. I also need to thank Simon Tse, who appeared on Talking Kotlin with me, and who wrote the first line of Kotlin at Hootsuite. Finally, my team at Hootsuite for driving Kotlin Culture with me, both within and outside Hootsuite.

When the slides and video of my talk are posted by JetBrains, I will share them here in another post!

The Cost of Kotlin Language Features

The next talk after mine was about the cost of Kotlin features relative to Java by Duncan McGregor.

The talk was very enlightening and despite there being a measurable cost to some of the features of Kotlin, null safety, let blocks, etc. Duncan’s conclusion is that “Everything is reassuringly OK”. So based on his research, keep using your fancy Kotlin language features.

Going Serverless with Kotlin

Next was a talk that I knew I had to attend from the title alone by Marcos Placona. Paul Cowles and I wrote on our experience using AWS Lambda functions for cross platform iOS and Android development earlier this year. That blog post formed the conference talk that I gave at AnDevCon in July of this year.

Marcos had experimented with a variety of cloud providers which was interesting as we have just used AWS Lambda and API Gateway here at Hootsuite. One of his points aligned with what we found, Serverless functions are very cheap to execute and the free tiers from cloud providers is generous.

I chatted with Marcos shortly after his talk about Serverless and about the open source library that we created at Hootsuite to help people set up and maintain Lambda functions with Terraform and Gradle. Our function currently uses Java, he inspired me to cut a ticket to convert it to Kotlin.

Two Stones, One Bird: Implementation Tradeoffs

In the final talk of the conference, Christina Lee talked about tradeoffs in implementation while reviewing some Kotlin language features.

She discussed some of the differences between: let, with, run, apply, and also. When I see these features discussed I think of the graphic that we created at Hootsuite last year to help us understand the symmetry of these functions. I’ve since seen a similar illustration in other places (a missed opportunity to work out loud on my part) but I will also share it here.

apply and also return the object that the functions are called on while run and let return the last expression of the lambda.

apply and run have ‘this’ as the receiver inside the lambda while also and let have the renamable ‘it’.

The symmetry of these functions makes them easier to understand and we have cases we each at Hootsuite described in our style guide.

Closing Panel

All good things…

The closing panel was very insightful and the panel answered a lot of hard questions. There were some laughs when Andrey was asked what else he might add to Kotlin. He rephrased the question as “What haven’t I stolen yet?” and gave immutable data as a possible answer.

It’s at the panel that I learned there are around 50 people working full time on the Kotlin language, no wonder they’re making such great strides with Kotlin/Native, multiplatform, and other initiatives! I’m very happy that Jetbrains has invested and continues to invest so much in the Kotlin language.

I also learned that 17% of all apps in the Google Play Store are using Kotlin! I’m sure that number will rise before the next KotlinConf. The conference app, on web, Android, and iOS was programmed in Kotlin using Kotlin/Native, Kotlin/JVM, and Kotlin/JS.

Andrey and Stephanie Cuthbertson also talked about the language foundation which was promised when Kotlin was made an official Android development language during Google I/O this year. They currently have a language committee and they are committed to developing Kotlin in a thoughtful manner. The committee so far has: William Cook, University of Texas at Austin, Andrey Breslav, JetBrains, Jeffrey van Gogh, Google.

Andrey reiterated that with Kotlin, he’s more concerned about the features that are not added than the features that are. He doesn’t want the language to become too large, if they add many new features in the future, it will likely be coupled with the gradual phasing out of other features.

Conclusion

After two whirlwind days the first KotlinConf is over! I can’t overstate how honored I was to be selected as a speaker and how happy I was to simply attend. I hope that I can continue to #workoutloud and share my journey with Kotlin and more!

If anyone would like to reach out to me about my talk, about growing Kotlin culture, or about the Atlas and Peon tools that we use at Hootsuite, please contact me at neil.power@hootsuite.com or on Twitter.

Datalab is an internal analytics team that powers data-driven decisions at Hootsuite. We build internal tools, data pipelines, and warehouses for other teams and analysts around the company. When I started on datalab and got introduced to all the technologies we are using here, one of my question was “Why do we have multiple databases? Why can’t we just use one?” especially after I learnt that a lot of the data is duplicated to both our Amazon Redshift cluster and MySQL database. There are numerous different databases out there: from MongoDB, MySQL, SQLite to Cockroach DB and Cassandra, and sometimes a combination of different DBMS can benefit your workflow and boost your productivity. So maybe you don’t really have to choose between Row and Column store!

On datalab we mainly use MySQL as the authoritative source of truth and Amazon Redshift, mostly, as read-only analytical data warehouse. In this blog post, I will try to show you how we use them together, how that benefits our productivity, and what are the differences you should be aware of, if you’re thinking of integrating either one (or both) into your workflow.

 

Column Store vs Row Store

The two most common types of databases are row-oriented and column-oriented systems. They have fundamentally different designs which make them efficient in different use cases.

On datalab, as you might guess, we deal with lots of data. We have dedicated analysts who perform all kinds of manipulations on data we collect from all our products to draw big pictures of products usage and perform detailed analyses on particular features and metrics for data driven decisions. To allow our analysts perform those complicated analyses, we have engineers on the team who built services and ETLs to store, update, and transfer the data between our databases. In order to optimize the work we do, we are using not one, but two data warehouses: a row-store and a column-store. In our case, MySQL and Amazon Redshift, respectively.

MySQL, a row based database, was our first data warehouse. It’s been around since 1995 and has become the world’s most popular open source database. As a traditional row oriented DBMS, it stores data as rows on the disk, meaning that all columns of a table are stored together and, if you are to access or alter any row of your table it could be done very easily and efficiently. Because of how records are stored on the disk (row by row, remember?) it is very fast to fetch entire rows.

Let’s imagine we have a user table with the following columns: userid, username, email, address, country, region, joindate, and an index key on joindate, it would look like this:

userid      username      email  address country     region join_date
10034 David Kim dkim@mail.com      NULL Canada North America .       2014-08-02
10035 Marie Joy NULL 1723 Secret Garden Rd     France Europe 2012-09-13
…. ….. …. ….. …. …. ….
 

Now, if we want to find all users that joined Hootsuite after June 2014 we would run the following query:

SELECT * FROM hsusers WHERE joindate >= ‘2017-07-01’ ;

It would be fast, assuming an index on join_date.  Likewise, inserting, updating, or deleting individual rows in this table would be very quick. Say that you wanted to update the address of just a few users, that would also be fast since all the user’s information is stored in one place, you’d only have to alter a few tuples on the disk.

However, these are not always the queries we use. A lot of tables are very sparse with numerous fields in them and we don’t necessarily need to access all the information. Now let’s say that in addition to our user table we now have a table that stores information about events and user payments.

Events table:

event user_id         timestamp
Message Sent         10035 2017-04-26 22:50:30
Image Viewed 10034 2017-03-26 18:48:37
…. ….. …..
 

Payments table:

userid       amount       currency     paymentdate member_plan
10034 12.00 CAD 2017-04-28 22:50:30      3
10035 134.13 CAD 2017-04-28 10:01:45 1
…. ….. …. ….. ….
 

Now we want to find all users who made payments in the last two years and had a low product usage rate. The database would have to join all those tables (user table, events, and payments), and access all the fields, even the ones that won’t be related to the final result. The tables need to be joined on user_id and then the final result could be returned. This can get inefficient with scaling up the data, and performing complex queries.

Accessing individual columns without touching the whole row is almost impossible with row-oriented architecture, since under the hood the database would likely have to read entire records in both tables.

This is when Amazon Redshift, comes in play!

Column-oriented  databases have been around since late 1960s as a niche technology, but have become dramatically popular for analytical workloads only in the last 5-10 years. These type of databases have advantages of data compression, late materialization of tuples, block iteration, code pipelining, and cache locality which makes them faster and more efficient for analytics types of queries. As you probably already guessed, data on the disk is stored as column by column and, as the result, accessing individual fields is extremely fast. So finding all users that made payments in the last two years and had a low product usage rate would be very fast in Redshift.

This is why Redshift is great for data analytics and that is why we use it as our data warehouse; very often we want to know things like, average user age, or revenue for X period of time, or correlation between plan and the size and industry of the organizations. Redshift is great for that!

Now, let’s try a more realistic example. Suppose, we still have our users and payments tables. And let’s imagine someone asked us to calculate monthly revenue of all Hootsuite users per plan. The following query would give us the results we want:

Running an analogous query on our production data with ~16 million records in the user table and ~18 million records in the payment table took 7 minutes to finish in MySQL and (ta-daa) 4.8 seconds in Redshift!

Yes, the difference is huge!

We already talked about the benefits of column-store for these kind of queries, but let’s dive a little bit deeper into what’s happening under the hood that makes Redshift in this case, so much more efficient.

First is late materialization: with late materialization of tuples, Redshift is able to effectively select needed columns, construct pairs and perform operations on the compressed data, before reconstructing that into the final tuples and returning the result. While MySQL would have to read all the tables, join them together, and perform needed operations.

Another benefit of Redshift system design is efficient computing. Redshift is a cluster based system, meaning that it consists of multiple nodes and computing can be distributed to different nodes in the cloud system in the way that would optimize the process the most.

But..

Well.. Redshift is not perfect for all queries. For the same design reasons it is not efficient for huge data updates or queries like the following:

In real life our user table would be pretty sparse, containing all sorts of information about our users. Since Redshift would store it by columns, it has to reconstruct the whole user table together by accessing individual columns that might even be stored on different machines and therefore involve sending information over the network to return the final result. This is a much slower process than fetching rows that are stored together in MySQL.

But this is not all..

Another thing that is good to know about Redshift is Vacuuming.

If you delete a record in Redshift it doesn’t actually get deleted from the disk, but instead gets marked as deleted or “dead”. Same thing with updates: the original record would be marked as deleted and, instead of modifying it, a new one is created with the updated information. You can imagine how much garbage data we would have after thousands and thousands of updates and deletes. This is what is vacuuming is used for: to get rid of all that garbage and free up the disk space. Another use case is making sure your records are always stored in order. If you specified a sort key on a table, your data would be sorted, but as you are inserting new records, Redshift doesn’t go and look for a proper place to insert that record into, but instead it just places it in the unsorted area on the disk, until you run Vacuuming that will sort that unsorted area and put the records into proper places for high efficiency. But the only catch there, is that it’s not done automatically, you have to schedule and perform vacuuming manually, which is one more thing to worry about.

This diagram illustrates how unsorted region of your table grows as new records get inserted

And everything is not that simple..

Let’s get back to our events table, and let’s say we have a sort key on the timestamp column. In this case, as new records get added, their timestamp is likely to be greater than the existing records’ timestamps. Say we had an outage and lost events data for the last four days (that would be a lot!). But we are lucky to have it somewhere else and be able to backfill the table. In this case, when we insert lost data into our events table and then perform vacuuming to make sure it’s sorted, only unsorted region would be overwritten since redshift knows that everything above is already sorted and vacuuming would be very fast.

If inserted data overlaps with existing data, then more regions of the table need to be overwritten. In this case, our new records are likely to have increasing timestamps, which will make our data be more or less in order, and would result in minimal overlap. But what if we also wanted to sort our data on userid, so that Redshift could efficiently find users? New records will have both very high and very low userid values. Now, data overlap will be much bigger, and if we insert many new records, vacuuming operation will need to re-write nearly entire table to keep all the records sorted on the disk by both timestamp and user_id:

 

And as the table grows, vacuuming costs will grow too, since costs of the merge step are proportional to the size of the table – we have one table that takes at least 6 hours to vacuum!  

There are strategies go around that reduce the costs of the problem: for example, if you have a huge table and you actually want to sort the data by both timestamp and user_id, you could split that table into smaller tables by timestamp ranges, and then create a view that would union those tables together. Then, when you insert a new data, it is likely that you will only ever need to modify the table with the most recent timestamp range, so you will only need to vacuum that small sub-table (much faster). But everything comes at a cost: this also means that your maintenance process will be more complex with a need for creating new tables as the dataset grows, switching vacuuming to those tables, update views, and having poor performance of certain types of queries.

Now, let’s get back to MySQL for a minute.

Another great thing about MySQL is the ability to validate and restrict the data. Redshift lacks many of these functions. One of the things I learnt the hard way was that Redshift UNIQUE key does not enforce actual uniqueness of records… you can define a UNIQUE key on a table, but it is more for you to be aware that that column is supposed to have unique member_id or name or something else, whereas MySQL would actually ensure that, and would not let you insert existing value into the table. The reason for this is, again, lack of traditional indexes in Redshift – it would be very inefficient for Redshift to check if a given value is unique, so there is no uniqueness constraints, and enforcing uniqueness is left to the application layer.

In other words, if you want to have some restrictions on your data, MySQL would do a great job. This is why a lot of the times we would load the data into MySQL first and then copy it over to our Redshift cluster for analysts to use. Luckily, we can dump data from MySQL to S3 quite efficiently, and bulk load it from S3 to Redshift blazingly fast, so this is actually a fairly simple and quick process.

If you are wondering why no one has come up with a database that would be something in the middle and would have benefits of both row and column based DBMS, I would recommend reading this paper that discusses the fundamental architectural differences between those two and proves that it is impossible for a row-oriented database to gain all the advantages of column-store while applying a number of optimizations.  The long and short of it, though, is that storing data as rows vs. columns, both on disk and in memory, comes with largely inescapable tradeoffs – the different designs will always have their pros and cons.

In the meantime, Hootsuite, as a lot of other companies, use a combination of databases for analytics. Our Redshift cluster serves primarily as a query layer, allowing us to perform analytical queries and generate reports relatively fast, and our MySQL database lets us validate data, and do work requiring regular inserting, updating and deleting of individual records (or small batches). When developing new ETLs and services we might use both Redshift and MySQL depending on the type of queries they would require. But generally speaking, having that flexibility and advantages of both DBMS, allows our team to be more productive without waiting hours to retrieve certain data; and we find it worth the overhead of copying data from one system to another.

About the Author

Daria is a co-op student on DataLab. She is in the Computer Science program with Software Engineering option at University of Victoria. In her spare time she likes drinking good coffee, traveling, and capturing nature with her camera.

 

 

 

 

 

Background

  All of our infrastructure at Hootsuite runs on AWS. We use thousands of AWS EC2 instances to power our microservices. EC2 instances are Amazon’s offering of resizable virtual servers hosted on the cloud. Particular groupings of these servers often share a virtual private network.

  Some of our servers expose public endpoints to the world, creating an attack vector. Let’s say a hacker succeeds in breaching one of these endpoints. The immediate problem is that they can gain access to the server, but the bigger problem is that they can also gain access to everything on the network, including other critical AWS resources. How can we preempt this exploitation?

  Fortunately Amazon’s security isn’t that short-sighted. They have IAM roles to prevent this from happening. What are IAM roles? An IAM (Identity and Access Management) role is a set of permissions that can be attached to entities such as accounts and servers to grant (and implicitly, limit) access to AWS resources. Since all API requests must be signed with AWS credentials, IAM roles are a flexible way of assigning these credentials.

  Excellent, this means can create some IAM roles, attach them to our servers, and trust that this will give us tight control over who can access what from where. This should stop a hacker who has breached our server’s security from exploiting that position to cause further damage elsewhere.

  In this diagram, the blue IAM role grants full access to the blue S3 bucket. We want to allow API requests coming from blue services to be able to access sensitive internal data while blocking requests from less secure services. We do this by attaching the IAM role to the node that the single blue service here lives on, Node 1. This privileges the billing service while restricting the exposed web service, since Node 2 does not have an IAM role that will give it the proper permissions to the bucket. Now we have what we wanted: the required credentials assigned to privileged services and refused to others.

  Not so fast. At Hootsuite, our services are not tied down to specific nodes. They live in containers — highly portable, lightweight, self-contained software packages. These containers are orchestrated by Kubernetes.

  Kubernetes is a clustered container orchestration platform. It has its own scheduling algorithm that determines which nodes containers get scheduled on based on a variety of factors. While we can dictate some aspects of this scheduling, we don’t oversee the whole process. As a result, we can’t always predict how containers will move around on our cluster. The fact that containers, and the services that run in them, are not tethered to specific nodes in a one-to-one fashion throws a wrench into our security model. The diagram shown earlier actually looks more like this:

  What is going on here? We still have the blue IAM role and the corresponding blue S3 bucket, but now we have two containers running on the same node. Only the billing service should have access to the bucket, yet if we attach the IAM role to its node, we risk granting the exposed service equal access. On the other hand, if we don’t attach the IAM role, we impair the billing service from reading or writing updates to the bucket. The consequences of both choices are unacceptable, presenting a conundrum. The previous diagram illustrated a many-to-one relationship between containers and the respective nodes, but a many-to-many representation is closest to modelling a real cluster:

  On a real cluster, blue and red services are distributed across any number of nodes, with nodes hosting any number of services. Since these services have varying permissions to resources, it makes no sense to funnel all their requests through a shared IAM role — a single set of permissions — just because they share a node.

  It would be great if we could attach multiple roles to a node and switch between them as needed. However, as the AWS documentation explains, “Note that only one role can be assigned to an EC2 instance at a time, and all applications on the instance share the same role and permissions.”

  This is the heart of the issue. Our Kubernetes clusters have multiple containers running on the same nodes with different access privileges. Moreover, these containers move around continuously and unpredictably across nodes. We need fine-grained and adaptable credential management to secure access of services on our distributed system. AWS’s typical solution of using IAM roles falls short of our security needs. This project was undertaken to find a solution to this problem.

Kube2IAM

  We wanted to solve this problem without eschewing IAM roles altogether. It would have been possible to sign services’ API requests without them, but given the advantages that IAM roles provided and their integration in the AWS ecosystem, it seemed logical to continue using them.

  The ideal solution would fix the deficiency of IAM roles by making a simple adjustment. Recall that by design, IAM roles can only be attached to entities such as accounts and servers. What if we could attach IAM roles to an additional entity — containers? This IAM-per-container idea is not novel, and after doing some research, it was clear that other companies were also thinking of this approach. There were a few open-source projects that showed potential and caught our attention. Among them was Lyft’s metadataproxy and Atlassian director Jerome Touffe-Blin’s kube2iam.

  At first, the metadataproxy seemed to do exactly what we wanted. It provided a mechanism by which containers could each be assigned an IAM role, and when services in those containers made API requests, they would obtain credentials exclusive to their roles. Sharing a node did not equate to sharing permissions. The problem with the metadataproxy, as we discovered, was not in its approach but in its implementation. The metadataproxy only supported the Docker client and Docker networking model and so made incorrect assumptions about our network layout, particularly when it came to container IP addresses. To use it we would have to significantly extend the code.

  Not finding success with that open-source project, we turned to the next: Kube2IAM. Kube2IAM was intended for Kubernetes clusters, as advertised in the name. It overcame the issues of metadataproxy by supporting different networking overlays that followed the Kubernetes standards. Reading the documentation, we recognized that its fundamental approach did not differ from metadataproxy’s. Their solution was to:

“redirect the traffic that is going to the EC2 metadata API to a [Kube2IAM] container running on each instance. The Kube2IAM container will make a call to the AWS API to retrieve temporary credentials and return these to the calling container. Other calls [that don’t require credentials] will be proxied to the EC2 metadata API.”

  There are a few unfamiliar terms here. The EC2 metadata API is what services must query to obtain credentials to sign their API requests. As stated earlier, all services need AWS credentials to make API requests. Credentials are enclosed in instance metadata (other metadata examples are hostnames, public keys, subnets), and this metadata is acquired by curling a static endpoint http://169.254.169.254/latest/meta-data/.

  Kube2IAM doesn’t want services to be able to directly query for their credentials. Instead it wants to act as a proxy that regulates metadata requests. That’s why the description notes that traffic to the metadata API should be redirected to a container managed by Kube2IAM itself. Thus the Kube2IAM container acts as a credential authority.

  One important point not mentioned in the description above is how the Kube2IAM container will “retrieve temporary credentials” for each calling container. It does this via an AWS API action AssumeRole. The AssumeRole action allows “existing IAM users to access AWS resources that they don’t already have access to.” When called with a valid IAM role, it assumes that role and returns a set of temporary security credentials for that role.

  If this sounds confusing, that’s understandable. Picture a special (gold) IAM role that can’t access any resource on the network. It is special by virtue of one property only: it can assume other roles that “trust” it. If the other roles have greater privileges, the gold IAM role temporarily elevates its own privilege by assuming those roles. Once it does, it can return the credentials for the roles it assumed.

  The following diagram displays the Kube2IAM container depicted as a gold diamond. For the sake of visual clarity, the gold diamond was drawn below the node and given a different colour/shape from the other services. In reality, the Kube2IAM is just an ordinary container running alongside other containers on the node. It is also important to note that the gold IAM role is attached to the node itself, not the container. Since it is the only role attached to the node, we aren’t breaking AWS rules.

  Each service is given a role annotation when it is deployed. The services in the diagram are given the annotations “red IAM role” or “blue IAM role” based on what access permissions they should have. Both these roles “trust” the gold IAM to assume them and return their credentials. When a metadata API request from a red service is made, Kube2IAM reads its role annotation and assumes the red role, returning credentials exclusive to red services. A hacker cannot improve the role annotation of the service they have hacked once it is already deployed without being a cluster administrator.

  Having a single role that can assume other roles and return credentials depending on the requester gives us the fine-grained, adaptable credential management we need to secure services’ access to resources on our network. Thus it solves our original problem.

Setup

  Detailed setup instructions can be found on the Kube2IAM docs. To underline the main points, Kube2IAM runs as a daemonset in Kubernetes, so one “instance” of the Kube2IAM container will run on every node and be restarted if killed. There were a few implementation challenges. First, we were confused about how AssumeRole worked and how to define trust policies. Figuring out how to evaluate whether Kube2IAM was properly working was not straightforward either. We made some assumptions about its behaviour that weren’t true and spent too long debugging the right results for a wrong test.

Conclusion

  Kube2IAM is a solid add-on for Kubernetes clusters running on AWS. It secures the network by granting containers IAM roles and enforcing their access privilege through role annotations. On a personal note, implementing this PoC provided a great opportunity to learn about security and networking.

About the Author

Nafisa Shazia is an Operations Developer Co-op on the Kubernetes Initiatives team. She studies Computer Science at UBC.

Positioning in CSS can be a nightmare, especially when you want to make all your pages responsive and look pretty on phones, tablets, laptops, and bigger screens. If we wanted to create layouts for an entire page, one option is to use a grid like Bootstrap. But what if we wanted to create layouts for something smaller, like items in a component?

As part of the WebDev team, I worked on several components that were used by editors in our CMS to create content for the Hootsuite website. This meant I was given a design of a component that was to be used, and it was my job to create it, considering responsiveness, translations, and content variability at all times. I was fairly comfortable with using the basics of CSS, and the transition to using SCSS at Hootsuite didn’t turn out to be a huge challenge either. What I struggled with the most was positioning – even tasks that seemed to be easy didn’t end up being as easy as they appeared.

Pre-Flexbox: Absolute Positioning

Let’s take a simple situation as an example: vertically centering something. Besides the horrendous idea of hard-coding the pixel values, a neat trick that would do the job would be to use absolute positioning:

See the Pen Vertical Centering – Absolute Positioning by Jieun Lee (@jieun-lee) on CodePen.

This isn’t a terrible idea, since it accomplishes the task with a few lines of code and is also fairly responsive, as the object would be centered at different screen sizes as well. However, it’s not the best solution, as it makes code overly complicated when we get to more complex situations, such as when using nested components and handling varying designs for different screen sizes. I kept thinking that there had to be a better way to approach these problems, and it turned out the solution I was looking for was flexbox.

The Transition to Using Flexbox

During a code review, one of my teammates suggested using flexbox to refactor my component. At first, it seemed like a lot of work because I would have to rewrite a lot of CSS after I spent a lot of time getting all the images and content to align nicely using absolute positioning. However, after a quick reading of flexbox from some articles I found online, it didn’t seem so bad. In fact, it seemed to make the tasks that should have been easy actually easy.

After some thought, I decided to start fresh with a blank CSS file – I literally scrapped all the CSS I had and re-wrote everything using flexbox. It was amazing to see how quickly I was able to re-create the component, and my file went from 150 lines to 100 lines. Refactoring was definitely worth it – of course, it did take a bit more time to redo the component, but it took much less time to implement and my code was shorter, cleaner, and easier to read and manage.

So let’s go back to that vertical centering problem. This is the same scenario created using flexbox:

See the Pen Vertical Centering – Flexbox by Jieun Lee (@jieun-lee) on CodePen.

All we had to do was apply display: flex; then set the vertical margin to auto. This was a much easier and more flexible solution.

Flexbox Layouts with an Example

There are many resources out there that explains the fundamentals of flexbox and its basic properties, so instead of being repetitive I will show you a sample layout built with flexbox and break it down into parts of how it was created. Let’s say we’re trying to make something like this:

See the Pen Flexbox Example – Final by Jieun Lee (@jieun-lee) on CodePen.

It’s a basic usage example, but it contains several usages of flexbox:

  • Overall Layout: the sidebar and main section are in a wrapper div, and the wrapper and header are in a page div
  • Header: the box that contains the title is flexible and fills the space that isn’t filled up by the circular buttons on the right
  • Sidebar: the menu items are spaced apart evenly from one another (this might look ugly on mobile, normally we wouldn’t keep the menu as a sidebar on smaller screens)
  • Main: contains three streams, evenly spaced, with each stream containing multiple text boxes with varying lengths (two streams on smaller screen sizes; this would normally be styled nicer on real components)

NOTE: If you’re not familiar with SCSS, part of the syntax may be confusing, so here’s a short guide to the SCSS syntax that you’ll encounter in my code:

  • Using variables, named using the dollar sign
  • (ex) $menu-color: #333;

  • Using nesting with ampersands (the ‘&’ replaces the parent in this case)
  • (ex) .header { &__main { //css }} would compile out to .header__main { //css }

Making the Menu Items Space Evenly on the Sidebar

Making items space evenly is easy with flexbox; simply use flexbox by adding display: flex; to the flex container and set margin: auto; to the flex items. This is the same technique we used to center a single item. Try changing the height to verify that it is flexible and evenly spaced at various height settings.

See the Pen Flexbox Example – Sidebar by Jieun Lee (@jieun-lee) on CodePen.

Splitting the Main Section into Streams

This is actually pretty simple, since all three streams (two, if you’re on mobile) have the same width, height, and flex properties. We will be able to see the desired behavior as soon as we add display: flex; and flex-direction: row;. However, the underlying property that makes this all work is set on the flex items (the streams).

In the code below, I have explicitly included flex: 0 1 auto; which is the default value in most modern browsers. The values represent flex-grow, flex-shrink, and flex-basis, respectively. None of the streams will grow larger than the specified width (flex-grow: 0;), but all of them have the equal ability to shrink (flex-shrink: 1;). They also have a flex-basis: auto; which is used to set the default size of the stream. This method is better than specifying widths in percentages, as it will be easier to make changes if we ever have to add or remove streams but still want to fill the entire space.

See the Pen Flexbox Example – Main Section by Jieun Lee (@jieun-lee) on CodePen.

Filling Remaining Space in the Top Header

The header is composed of three circular buttons with fixed sizes on the right and a titlebar on the left that fills the space that isn’t taken up by the buttons. When the width is adjusted, the titlebar shrinks or grows accordingly, and the title text inside centers itself automatically (just another use of flexbox with auto margin).

The only actual work we have to do is set header__menu to have flex-shrink: 0; instead of the default value of 1. This means these menu items cannot shrink, so the titlebar with an initial width of 100% shrinks instead, as it still has flex-shrink: 1;.

See the Pen Flexbox Example – Header by Jieun Lee (@jieun-lee) on CodePen.

Combining the Sections

Basically, the “page” div holds the header and the wrapper, and the “wrapper” holds the sidebar and the main section. Of course, I used display: flex; on both the page and the wrapper. This example uses minimal content (no menus, only one stream, etc.) to illustrate the layout.

As mentioned previously, the page div holds the header and the wrapper (with a flex-direction: column;). We want the header to be a fixed height, which we specify. We also add flex-shrink: 0; so that the heading section will never get squished. This means the wrapper div (below the header) has a flex-shrink: 1; so it will adjust its height as needed.

The wrapper works in a similar way; it has a flex-direction: row; and a sidebar with a fixed size. For the sidebar we set width: 20%; and flex-shrink: 0; and have the main section have a width: 100% so it can shrink the width as needed.

See the Pen Flexbox Example – Layout by Jieun Lee (@jieun-lee) on CodePen.

Put these four steps together to get the final product that was illustrated at the top of this section. If all of this seems easy to you, try and create this exact scenario without using flexbox – in addition to looking correct, it has to be scalable to content changes and different screen sizes. You’ll quickly realize how much flexbox simplifies the whole process.

One final note about flexbox is that there are some older browsers that have limited or no support for flexbox. For example, Internet Explorer 10 has a different default value for flex, and some properties such as flex-shrink are not supported. However, most newer browsers do have full support for flexbox, but we may have to add prefixes (ex. display: -webkit-flex;) to make sure it is compatible. You may be able to find an automated way to do this, or use something like SCSS mixins to add flex properties all at once.

There’s a lot of neat things you can do with flexbox, also by using the properties I didn’t have in my examples. Either way, I encourage you to try it out – you’ll be amazed by how easily you can create interesting layouts with just a few short lines of code.

About the Author

Jieun Lee is a Software Developer Co-op on the WebDev Team. Jieun studies Computer Science and Mathematics at UBC, and likes to play piano in her spare time. Connect with her on LinkedIn.

Loading ...