Posts from September 2017

Background

  All of our infrastructure at Hootsuite runs on AWS. We use thousands of AWS EC2 instances to power our microservices. EC2 instances are Amazon’s offering of resizable virtual servers hosted on the cloud. Particular groupings of these servers often share a virtual private network.

  Some of our servers expose public endpoints to the world, creating an attack vector. Let’s say a hacker succeeds in breaching one of these endpoints. The immediate problem is that they can gain access to the server, but the bigger problem is that they can also gain access to everything on the network, including other critical AWS resources. How can we preempt this exploitation?

  Fortunately Amazon’s security isn’t that short-sighted. They have IAM roles to prevent this from happening. What are IAM roles? An IAM (Identity and Access Management) role is a set of permissions that can be attached to entities such as accounts and servers to grant (and implicitly, limit) access to AWS resources. Since all API requests must be signed with AWS credentials, IAM roles are a flexible way of assigning these credentials.

  Excellent, this means can create some IAM roles, attach them to our servers, and trust that this will give us tight control over who can access what from where. This should stop a hacker who has breached our server’s security from exploiting that position to cause further damage elsewhere.

  In this diagram, the blue IAM role grants full access to the blue S3 bucket. We want to allow API requests coming from blue services to be able to access sensitive internal data while blocking requests from less secure services. We do this by attaching the IAM role to the node that the single blue service here lives on, Node 1. This privileges the billing service while restricting the exposed web service, since Node 2 does not have an IAM role that will give it the proper permissions to the bucket. Now we have what we wanted: the required credentials assigned to privileged services and refused to others.

  Not so fast. At Hootsuite, our services are not tied down to specific nodes. They live in containers — highly portable, lightweight, self-contained software packages. These containers are orchestrated by Kubernetes.

  Kubernetes is a clustered container orchestration platform. It has its own scheduling algorithm that determines which nodes containers get scheduled on based on a variety of factors. While we can dictate some aspects of this scheduling, we don’t oversee the whole process. As a result, we can’t always predict how containers will move around on our cluster. The fact that containers, and the services that run in them, are not tethered to specific nodes in a one-to-one fashion throws a wrench into our security model. The diagram shown earlier actually looks more like this:

  What is going on here? We still have the blue IAM role and the corresponding blue S3 bucket, but now we have two containers running on the same node. Only the billing service should have access to the bucket, yet if we attach the IAM role to its node, we risk granting the exposed service equal access. On the other hand, if we don’t attach the IAM role, we impair the billing service from reading or writing updates to the bucket. The consequences of both choices are unacceptable, presenting a conundrum. The previous diagram illustrated a many-to-one relationship between containers and the respective nodes, but a many-to-many representation is closest to modelling a real cluster:

  On a real cluster, blue and red services are distributed across any number of nodes, with nodes hosting any number of services. Since these services have varying permissions to resources, it makes no sense to funnel all their requests through a shared IAM role — a single set of permissions — just because they share a node.

  It would be great if we could attach multiple roles to a node and switch between them as needed. However, as the AWS documentation explains, “Note that only one role can be assigned to an EC2 instance at a time, and all applications on the instance share the same role and permissions.”

  This is the heart of the issue. Our Kubernetes clusters have multiple containers running on the same nodes with different access privileges. Moreover, these containers move around continuously and unpredictably across nodes. We need fine-grained and adaptable credential management to secure access of services on our distributed system. AWS’s typical solution of using IAM roles falls short of our security needs. This project was undertaken to find a solution to this problem.

Kube2IAM

  We wanted to solve this problem without eschewing IAM roles altogether. It would have been possible to sign services’ API requests without them, but given the advantages that IAM roles provided and their integration in the AWS ecosystem, it seemed logical to continue using them.

  The ideal solution would fix the deficiency of IAM roles by making a simple adjustment. Recall that by design, IAM roles can only be attached to entities such as accounts and servers. What if we could attach IAM roles to an additional entity — containers? This IAM-per-container idea is not novel, and after doing some research, it was clear that other companies were also thinking of this approach. There were a few open-source projects that showed potential and caught our attention. Among them was Lyft’s metadataproxy and Atlassian director Jerome Touffe-Blin’s kube2iam.

  At first, the metadataproxy seemed to do exactly what we wanted. It provided a mechanism by which containers could each be assigned an IAM role, and when services in those containers made API requests, they would obtain credentials exclusive to their roles. Sharing a node did not equate to sharing permissions. The problem with the metadataproxy, as we discovered, was not in its approach but in its implementation. The metadataproxy only supported the Docker client and Docker networking model and so made incorrect assumptions about our network layout, particularly when it came to container IP addresses. To use it we would have to significantly extend the code.

  Not finding success with that open-source project, we turned to the next: Kube2IAM. Kube2IAM was intended for Kubernetes clusters, as advertised in the name. It overcame the issues of metadataproxy by supporting different networking overlays that followed the Kubernetes standards. Reading the documentation, we recognized that its fundamental approach did not differ from metadataproxy’s. Their solution was to:

“redirect the traffic that is going to the EC2 metadata API to a [Kube2IAM] container running on each instance. The Kube2IAM container will make a call to the AWS API to retrieve temporary credentials and return these to the calling container. Other calls [that don’t require credentials] will be proxied to the EC2 metadata API.”

  There are a few unfamiliar terms here. The EC2 metadata API is what services must query to obtain credentials to sign their API requests. As stated earlier, all services need AWS credentials to make API requests. Credentials are enclosed in instance metadata (other metadata examples are hostnames, public keys, subnets), and this metadata is acquired by curling a static endpoint http://169.254.169.254/latest/meta-data/.

  Kube2IAM doesn’t want services to be able to directly query for their credentials. Instead it wants to act as a proxy that regulates metadata requests. That’s why the description notes that traffic to the metadata API should be redirected to a container managed by Kube2IAM itself. Thus the Kube2IAM container acts as a credential authority.

  One important point not mentioned in the description above is how the Kube2IAM container will “retrieve temporary credentials” for each calling container. It does this via an AWS API action AssumeRole. The AssumeRole action allows “existing IAM users to access AWS resources that they don’t already have access to.” When called with a valid IAM role, it assumes that role and returns a set of temporary security credentials for that role.

  If this sounds confusing, that’s understandable. Picture a special (gold) IAM role that can’t access any resource on the network. It is special by virtue of one property only: it can assume other roles that “trust” it. If the other roles have greater privileges, the gold IAM role temporarily elevates its own privilege by assuming those roles. Once it does, it can return the credentials for the roles it assumed.

  The following diagram displays the Kube2IAM container depicted as a gold diamond. For the sake of visual clarity, the gold diamond was drawn below the node and given a different colour/shape from the other services. In reality, the Kube2IAM is just an ordinary container running alongside other containers on the node. It is also important to note that the gold IAM role is attached to the node itself, not the container. Since it is the only role attached to the node, we aren’t breaking AWS rules.

  Each service is given a role annotation when it is deployed. The services in the diagram are given the annotations “red IAM role” or “blue IAM role” based on what access permissions they should have. Both these roles “trust” the gold IAM to assume them and return their credentials. When a metadata API request from a red service is made, Kube2IAM reads its role annotation and assumes the red role, returning credentials exclusive to red services. A hacker cannot improve the role annotation of the service they have hacked once it is already deployed without being a cluster administrator.

  Having a single role that can assume other roles and return credentials depending on the requester gives us the fine-grained, adaptable credential management we need to secure services’ access to resources on our network. Thus it solves our original problem.

Setup

  Detailed setup instructions can be found on the Kube2IAM docs. To underline the main points, Kube2IAM runs as a daemonset in Kubernetes, so one “instance” of the Kube2IAM container will run on every node and be restarted if killed. There were a few implementation challenges. First, we were confused about how AssumeRole worked and how to define trust policies. Figuring out how to evaluate whether Kube2IAM was properly working was not straightforward either. We made some assumptions about its behaviour that weren’t true and spent too long debugging the right results for a wrong test.

Conclusion

  Kube2IAM is a solid add-on for Kubernetes clusters running on AWS. It secures the network by granting containers IAM roles and enforcing their access privilege through role annotations. On a personal note, implementing this PoC provided a great opportunity to learn about security and networking.

About the Author

Nafisa Shazia is an Operations Developer Co-op on the Kubernetes Initiatives team. She studies Computer Science at UBC.

Positioning in CSS can be a nightmare, especially when you want to make all your pages responsive and look pretty on phones, tablets, laptops, and bigger screens. If we wanted to create layouts for an entire page, one option is to use a grid like Bootstrap. But what if we wanted to create layouts for something smaller, like items in a component?

As part of the WebDev team, I worked on several components that were used by editors in our CMS to create content for the Hootsuite website. This meant I was given a design of a component that was to be used, and it was my job to create it, considering responsiveness, translations, and content variability at all times. I was fairly comfortable with using the basics of CSS, and the transition to using SCSS at Hootsuite didn’t turn out to be a huge challenge either. What I struggled with the most was positioning – even tasks that seemed to be easy didn’t end up being as easy as they appeared.

Pre-Flexbox: Absolute Positioning

Let’s take a simple situation as an example: vertically centering something. Besides the horrendous idea of hard-coding the pixel values, a neat trick that would do the job would be to use absolute positioning:

See the Pen Vertical Centering – Absolute Positioning by Jieun Lee (@jieun-lee) on CodePen.

This isn’t a terrible idea, since it accomplishes the task with a few lines of code and is also fairly responsive, as the object would be centered at different screen sizes as well. However, it’s not the best solution, as it makes code overly complicated when we get to more complex situations, such as when using nested components and handling varying designs for different screen sizes. I kept thinking that there had to be a better way to approach these problems, and it turned out the solution I was looking for was flexbox.

The Transition to Using Flexbox

During a code review, one of my teammates suggested using flexbox to refactor my component. At first, it seemed like a lot of work because I would have to rewrite a lot of CSS after I spent a lot of time getting all the images and content to align nicely using absolute positioning. However, after a quick reading of flexbox from some articles I found online, it didn’t seem so bad. In fact, it seemed to make the tasks that should have been easy actually easy.

After some thought, I decided to start fresh with a blank CSS file – I literally scrapped all the CSS I had and re-wrote everything using flexbox. It was amazing to see how quickly I was able to re-create the component, and my file went from 150 lines to 100 lines. Refactoring was definitely worth it – of course, it did take a bit more time to redo the component, but it took much less time to implement and my code was shorter, cleaner, and easier to read and manage.

So let’s go back to that vertical centering problem. This is the same scenario created using flexbox:

See the Pen Vertical Centering – Flexbox by Jieun Lee (@jieun-lee) on CodePen.

All we had to do was apply display: flex; then set the vertical margin to auto. This was a much easier and more flexible solution.

Flexbox Layouts with an Example

There are many resources out there that explains the fundamentals of flexbox and its basic properties, so instead of being repetitive I will show you a sample layout built with flexbox and break it down into parts of how it was created. Let’s say we’re trying to make something like this:

See the Pen Flexbox Example – Final by Jieun Lee (@jieun-lee) on CodePen.

It’s a basic usage example, but it contains several usages of flexbox:

  • Overall Layout: the sidebar and main section are in a wrapper div, and the wrapper and header are in a page div
  • Header: the box that contains the title is flexible and fills the space that isn’t filled up by the circular buttons on the right
  • Sidebar: the menu items are spaced apart evenly from one another (this might look ugly on mobile, normally we wouldn’t keep the menu as a sidebar on smaller screens)
  • Main: contains three streams, evenly spaced, with each stream containing multiple text boxes with varying lengths (two streams on smaller screen sizes; this would normally be styled nicer on real components)

NOTE: If you’re not familiar with SCSS, part of the syntax may be confusing, so here’s a short guide to the SCSS syntax that you’ll encounter in my code:

  • Using variables, named using the dollar sign
  • (ex) $menu-color: #333;

  • Using nesting with ampersands (the ‘&’ replaces the parent in this case)
  • (ex) .header { &__main { //css }} would compile out to .header__main { //css }

Making the Menu Items Space Evenly on the Sidebar

Making items space evenly is easy with flexbox; simply use flexbox by adding display: flex; to the flex container and set margin: auto; to the flex items. This is the same technique we used to center a single item. Try changing the height to verify that it is flexible and evenly spaced at various height settings.

See the Pen Flexbox Example – Sidebar by Jieun Lee (@jieun-lee) on CodePen.

Splitting the Main Section into Streams

This is actually pretty simple, since all three streams (two, if you’re on mobile) have the same width, height, and flex properties. We will be able to see the desired behavior as soon as we add display: flex; and flex-direction: row;. However, the underlying property that makes this all work is set on the flex items (the streams).

In the code below, I have explicitly included flex: 0 1 auto; which is the default value in most modern browsers. The values represent flex-grow, flex-shrink, and flex-basis, respectively. None of the streams will grow larger than the specified width (flex-grow: 0;), but all of them have the equal ability to shrink (flex-shrink: 1;). They also have a flex-basis: auto; which is used to set the default size of the stream. This method is better than specifying widths in percentages, as it will be easier to make changes if we ever have to add or remove streams but still want to fill the entire space.

See the Pen Flexbox Example – Main Section by Jieun Lee (@jieun-lee) on CodePen.

Filling Remaining Space in the Top Header

The header is composed of three circular buttons with fixed sizes on the right and a titlebar on the left that fills the space that isn’t taken up by the buttons. When the width is adjusted, the titlebar shrinks or grows accordingly, and the title text inside centers itself automatically (just another use of flexbox with auto margin).

The only actual work we have to do is set header__menu to have flex-shrink: 0; instead of the default value of 1. This means these menu items cannot shrink, so the titlebar with an initial width of 100% shrinks instead, as it still has flex-shrink: 1;.

See the Pen Flexbox Example – Header by Jieun Lee (@jieun-lee) on CodePen.

Combining the Sections

Basically, the “page” div holds the header and the wrapper, and the “wrapper” holds the sidebar and the main section. Of course, I used display: flex; on both the page and the wrapper. This example uses minimal content (no menus, only one stream, etc.) to illustrate the layout.

As mentioned previously, the page div holds the header and the wrapper (with a flex-direction: column;). We want the header to be a fixed height, which we specify. We also add flex-shrink: 0; so that the heading section will never get squished. This means the wrapper div (below the header) has a flex-shrink: 1; so it will adjust its height as needed.

The wrapper works in a similar way; it has a flex-direction: row; and a sidebar with a fixed size. For the sidebar we set width: 20%; and flex-shrink: 0; and have the main section have a width: 100% so it can shrink the width as needed.

See the Pen Flexbox Example – Layout by Jieun Lee (@jieun-lee) on CodePen.

Put these four steps together to get the final product that was illustrated at the top of this section. If all of this seems easy to you, try and create this exact scenario without using flexbox – in addition to looking correct, it has to be scalable to content changes and different screen sizes. You’ll quickly realize how much flexbox simplifies the whole process.

One final note about flexbox is that there are some older browsers that have limited or no support for flexbox. For example, Internet Explorer 10 has a different default value for flex, and some properties such as flex-shrink are not supported. However, most newer browsers do have full support for flexbox, but we may have to add prefixes (ex. display: -webkit-flex;) to make sure it is compatible. You may be able to find an automated way to do this, or use something like SCSS mixins to add flex properties all at once.

There’s a lot of neat things you can do with flexbox, also by using the properties I didn’t have in my examples. Either way, I encourage you to try it out – you’ll be amazed by how easily you can create interesting layouts with just a few short lines of code.

About the Author

Jieun Lee is a Software Developer Co-op on the WebDev Team. Jieun studies Computer Science and Mathematics at UBC, and likes to play piano in her spare time. Connect with her on LinkedIn.