Have you ever been blissfully stumbling on your iPad or mobile phone when you’ve found yourself wondering, “is there a wizard behind these stumbles?†We have many wizards at StumbleUpon, but today we want to highlight the Clients Team, which is in charge of our mobile apps, web bar, and extensions for Firefox and Chrome. The team took time out of their busy schedules to answer some questions together.

[ http://www.youtube.com/embed/zUwZwkioKX0 ]

Steven explains how he studied different swiping preferences in developing the “Swipe-to-Stumble” functionality on the recently-revamped StumbleUpon iPad app.

Favorite places to stumble on a mobile device:

In bed or on the bus.

Best parts about working on clients at StumbleUpon:

Clients are what people actually get to see. If you have a really nice car everyone sees the paint job, but no one asks about the engine. It makes this team’s work fun and easy to explain.

Another plus is working closely with every team. For example, Clients needs to talk to the Recommendations Team to fine-tune quality for specific devices, and to Infrastructure to fine-tune performance and speed. From a general quality perspective, they need to make sure they’re not showing content that a mobile user couldn’t access easily, like a Firefox extension on a mobile device or flash on an iPhone. Users have different expectations in terms of types of content, speed, and more depending on the kind of phone they prefer.

Another benefit is that StumbleUpon is very flat in terms of structure and organization. Developers constantly have conversations with other teams about overall architecture, and everyone works together on the full product. The team recalled several nights during the iPad app launch when engineers across all teams sat around talking about how a certain user interface should look. StumbleUpon CEO Garrett Camp spent hours exchanging ideas with developers from multiple levels and roles in these late-night chats.

Hardest challenges you’ve tackled recently:

The “Fight Gone Bad†workout routine at Crossfit, the gym across the street from the office. The Clients Team goes together twice a week. It’s a lot of Olympic weightlifting and short high-intensity workouts – sprinting, jump-roping, box jumps, dead lifts. They’re really sore afterwards.

Also, learning the ins and outs of graphic designs and mobile layouts, making web bar stumbling faster, coordinating the release of the iPad app.

Most recent stumbles:

The Girl Who Loves to Levitate (My Modern Met)
XKCD: Standards
How Hydroelectricity Works

Interested in joining the team? Apply now!

Stumble On,

Katie
Social Media Manager

People often ask me why I left a large, well-established tech company to work for StumbleUpon. Here’s what I tell them:

 

You Work in the Open

Big companies (like Google, Yahoo, Apple, etc.) tend to be rather secretive about what they do and the technology they build. In many cases, it makes sense since their technology is giving them a strong competitive advantage. The Silicon Valley is made of a myriad of small and medium-size startups, and through open-source projects, they implicitly “team up” to build tools and products that have a chance to stand up against the technology built by big companies. After joining StumbleUpon, I started to discover this ecosystem, going to various meetups and talking to people at other companies. This proved to be invaluable to learn new things and build new connections.

When you open-source your own projects, it’s also very rewarding to see other companies adopting it. At StumbleUpon we love open-source projects. We rely extensively on open-source technology such as Linux, MySQL, memcached, gearman, ZooKeeper, to name just a few. We’re one of the major companies supporting and iterating on HBase and we’re even helping other companies like Facebook or Adobe adopt and deploy it. It’s a fair game: we help these projects get off the ground, and eventually they help us in return by contributing engineering time back to the project. Recently I built OpenTSDB, our distributed monitoring system (don’t miss my upcoming talk on OpenTSDB  at OSCON Data on July 25, in Portland, OR).

You Wear Multiple Hats

One of the things I enjoy most about being an engineer at StumbleUpon is how I can work on many wildly different things. I’ve learned tons of things I could never have learned at my previous employer, because they had that problem solved already or because it wasn’t in my scope of responsibility to solve that issue. We have a very wide range of interesting problems to solve at StumbleUpon, and our engineers have a lot of latitude to pick their battles or have their say.

When we rebuilt our main datacenter, for instance, we had to solve a variety of issues in different areas, from server hardware to network design to front-end serving and load-balancing infrastructure, just to cite a few. We ended up designing and building a new infrastructure that relies exclusively on open-source technology (including OpenBSD, Varnish, and Perlbal), to replace enterprise gear like Cisco ASA firewalls or Citrix Netscalers (HTTP load balancers). We built a new, modern, highly performant, and efficient network using Arista gear, relying exclusively on open standards (such as VRRP or ECMP).

You Have the Right Resources

StumbleUpon has a rather interesting story, being a “re-startup.†This means that a lot of parameters are rather different from your typical Silicon Valley venture. It’s almost like taking the best part of being a startup without inheriting many downsides. Unlike small startups, we aren’t primarily constrained on resources or budget. It’s very exciting and enticing to work in an environment where you can change just about anything, help shape the company, and execute quickly, without the burden of the red tape typically found in larger organizations, and have the resources to execute on your ideas.

For a list of open engineering positions, check out our jobs page. And to see how we like to have fun at work, visit our Flickr stream!

Stumble On,

Benoît Sigoure
Site Reliability Engineer

At StumbleUpon we like to travel around the world to talk at conferences about the technologies we are working on, and this summer is going to be particularly busy:

Berlin Buzzwords, June 6-7 – At this conference, which focuses on open source software centered around scalable search, data-analysis in the cloud, and NoSQL-databases, I’ll present about the multiple applications of HBase (a topic we hear asked about a lot from HBase users). I’ll discuss the major ways StumbleUpon uses HBase, including live traffic serving and offline data crunching. Service level agreements (SLAs) are very different in both cases, and HBase has to be top notch. I will explore those two use cases and others by giving tips on how to configure your clusters, design your tables, and architect the database access layer.

O’Reilly OSCON Data, July 25-29 – Benoît Sigoure, Site Reliability Engineer, is going to this open source conference in July to talk about OpenTSDB, a real-time data monitoring program that he built himself and is catching on at several tech companies. It uses HBase to store billions of data points, such as CPU load, that are generated by our servers and applications.

I will also be part of OSCON Data to present StumbleUpon’s experience using Hive and HBase together. Hive is an Apache project that’s used as a layer of abstraction on top of MapReduce and HDFS so that our business analysts and engineers can query our big data sets by using a language that looks almost like SQL, without needing to write code. HBase is a distributed database that uses HDFS to store its data, and without it StumbleUpon couldn’t operate. As Hive’s support for HBase (i.e. running MR jobs on top of HBase) is rather recent, we wanted to tell other potential users what they should expect from this integration down the road.

If we are lucky enough to be part of the same conference as you, don’t be afraid to come up and ask us questions on these subjects. Just be aware that we are very passionate and might get a little carried away. Feel free to talk to us about open engineering positions at StumbleUpon as well; we’d be happy to give out more details.

Happy Hacking,

Jean-Daniel Cryans
Database Engineer, HBase committer

This is the first in a new series of posts where we’ll be regularly updating you on the cool stuff our engineers are busy working on. Feel free to ask questions in the comments section! And if you’re interested in developer positions we have available, check out our jobs page.

2011 – The Year of Data Growth (2.5 Billion Data Points a Week!)

Our engineering team has been working around the clock to manage and monitor increasing amounts of data as our userbase continues to grow past 15 million. We’re lucky to have a stellar team of HBase specialists – including Michael Stack, one of the creators of HBase (an Apache project), and Jean-Daniel “J-D†Cryans, database engineer and HBase committer – that helps StumbleUpon manage 2.5 billion data points every week. (Since there are only eight HBase committers in existence, Michael and J-D take their jobs very seriously!) We couldn’t manage our data without open source projects like HBase and Hadoop – they’re critical to our ability to build a fast and reliable user experience.

OpenTSDB: Debugging Downtimes

OpenTSDB is a program that utilizes HBase and was developed entirely in-house by our own Benoît Sigoure, Site Reliability Engineer. It helps us monitor second-by-second changes in our traffic and connectivity and detect spam and malicious activity as quickly as possible, all of which have been crucial as we expand to support more stumbling activity. Since Benoît first released OpenTSDB last September, several major companies have begun using it for their own data monitoring needs, finding it easy to throw any kind of data computation problem at it without issue.

 

The real-time, extremely granular data that OpenTSDB provides helps reduce the Mean Time To Resolution (MTTR) of outages. The program has brought our diagnosis time from hours to minutes, simply by giving us more visibility into second-by-second glitches. It’s even revealed problems we hadn’t noticed before or that were lying dormant until traffic spiked.

For example, recently we had a configuration problem on some of our databases that was causing a small fraction of our stumbles to be served extremely slowly (over 3 seconds, which is unacceptably slow for anyone waiting to see the best of the web). OpenTSDB helped us spot and resolve the issue because it doesn’t overlook even the smallest fraction of activity.

ElasticSearch: Making Data Easy to Find

StumbleUpon is using ElasticSearch, an open-source, scalable data search solution, to power some upcoming new features. Here’s one example of why we love it: say you wanted to retrieve web pages that you’d stored in HBase. To do this with HBase, you’d have to assign unique IDs to each web page and remember these IDs in order to retrieve them later. ElasticSearch can take a web page you’ve submitted and index every word on this page. All you need to do if you need the web page again is remember a keyword present on the page, and the technology will find what you need.

ElasticSearch also enables our engineering teams to work cross-functionally. Even engineers not trained on searching data sets like HBase can run functions using ElasticSearch technology. For example, our research team uses ElasticSearch when they conduct natural language processing tests to improve recommendation methods. Plus, it’s fast and easy to implement: I was able to integrate ElasticSearch in our analytics dashboard after just an hour.

Thanks for reading! If you want to read the more technical details of what’s on our engineers’ plates these days, head to our Developer’s Blog.

Happy Hacking,

Josh Eichorn
Senior Software Engineer

Heads-up – the 2011 Hadoop Summit is fast approaching! You won’t find a better gathering of straight up Hadoop tech talks and mingling with fellow Hadoopineers this side of the Mississippi.  This year promises to be bigger and more techy than ever before. And, by popular demand, keynotes have been purged from the agenda!

Owen O’Malley and I are co-chairs of the community track, and we’re looking for submissions on anything that pertains to the Hadoop community, including roadmaps, Hadoop contributions, development best practices, or community feeding and grooming. The deadline for abstract submissions is this Friday, May 6, 2011, so you’ll need to get a bit of a move on if you intend to present. Last I heard, the deadline is final and won’t be extended. If community isn’t your thing, consider submitting a talk to one of the other Summit tracks, Operations & Management or Applications & Research.

Stumble On,

 

 

 

 

Michael Stack
Senior Software Architect
Apache Software Foundation Member
Apache HBase PMC Member

 

 

 

At StumbleUpon’s headquarters in San Francisco on Tuesday night, about 70 engineers and HBase committers discussed the latest optimizations and utilizations of HBase, an open-source distributed database that simplifies managing and retrieving huge amounts of data. StumbleUpon relies on HBase to process all the data we receive on user preferences and stumbling patterns, and it helps us ensure our recommendations are accurate and high-quality.  Many large-scale web sites use HBase and Hadoop, the software framework on which it runs, to process jobs like this. On Tuesday, we heard from three HBase users who presented hacks they’ve created to get even more out of the architecture:

Todd Lipcon (twitter.com/tlipcon) from Cloudera, a software company that provides Apache Hadoop-based software and services, discussed how to avoid pauses, or moments when retrieving data is impossible, caused by the Java Garbage Collector. Todd explained that most garbage collection pauses are caused by fragmentation, and that MSLAB, a new memory allocator for HBase that plays well with the Garbage Collector, effectively moves all memory allocations into contiguous 2MB chunks.

Next, Benoit Sigoure, Site Reliability Engineer from StumbleUpon, talked about designing OpenTSDB, a distributed, scalable time series database. He discussed how OpenTSDB uses HBase to store and retrieve billions of highly granular data points in real-time and to present this data in custom graphs that even business folk like me can understand. The upshot:  No single point of failure and an intuitive interface for viewing data (like stumbles over a time period, for example). Check out his entire presentation here.

Finally, Darren Erik Vengroff from RichRelevance, which designs software for product recommendations one might see on Amazon and Netflix, talked about creating a language called BigQL, built on top of a new HBase Coprocessor feature, that data analysts who know SQL could understand. His aim was to give these data analysts a program that, in his words, is “easy to use, fits their way of thinking, and solves their problems, but that’s also tuned to the backend data stores we want to run on.â€

But I have to admit that my favorite part of the evening were the analogies that engineers have for certain program operations and scenarios. Here are some of the best from last night, in my opinion:

Besides getting a peek into the world of big data and imaginative engineering terms (not to mention munching on pizza and drinking beer from the StumbleUpon kegerator), I got the chance to chat with leading HBase developers like Ted Dunning, Chief Applications Architect at MapR Technologies. When I asked him about HBase’s place in the data tech community these days, Ted referenced this Arthur C. Clarke quote: “Sufficiently advanced technology is indistinguishable from magic.â€

“HBase is just beginning to grow up, to be magic in the sense that it hides the complexity,†he said. He added that the question that all new technology promoters face is the same: “How do you communicate a need to people who don’t know what they can’t do?â€

To find out more about these events and join the Bay Area HBase User Group, click here.

Stumble On,

Katie Gray
Communications Manager

Here at StumbleUpon, we’re always testing and iterating. Lately we’ve embraced Selenium, a popular open-source tool for developing automated tests of web sites. It’s helped us quickly test functionality such as advertiser registration, advertiser credit-card charges, su.pr scheduled posts, and StumbleThru on the web bar. That’s why we were excited to host the latest Selenium Meet-Up at our San Francisco office last Tuesday, when Kohsuke Kawaguchi gave a talk on “Hudson, Distributed Builds, and Selenium.â€

                                                                                                                (c) SauceLabs
MAKING QA EASIER Kohsuke Kawaguchi talks about Hudson, the open-source continuous integration tool he created, at the monthly Selenium Meetup hosted at StumbleUpon last Tuesday.

Kawaguchi is the creator of Hudson, a widely-used open-source tool for continuous integration (CI). In his presentation to 65 software QA engineers, he said that Hudson provides fast evaluation and reporting to engineering and QA teams. Whenever a developer commits a change to source code, Hudson can build the web site or application from that source, execute Selenium tests to determine if any functionality has been broken by the changed code, and immediately notify both teams of any problems found by the build and tests.

The San Francisco Selenium Meet-Up is held monthly at different San Francisco and Silicon Valley companies. Sponsored by Sauce Labs, these meetings start with schmoozing over pizza and drinks, followed by one or more formal talks on Selenium topics, and end with still more schmoozing. Click here to see where upcoming meetups will be held!

 

Hello everyone, my name is Ryan Rawson, a software engineer working on data storage at Stumbleupon.com. I am currently working on HBase, an open source distributed storage system, in which I am the committer on the project. HBase is part of the Hadoop ecosystem, and like the rest of Hadoop projects we take a distributed, low-power and cost-effective approach to storing data. Instead of purchasing one or two very large computers, HBase runs on a network of smaller and cheaper (both absolutely, and by dollars, per mips/ram/disk) machines. By distributing the data across multiple smaller machines, we get better performance since we can leverage additional resources merely by adding more machines. We also gain independence from any individual machine failure as a result.

In addition to actually writing code for HBase, I have presented about the technology publicly. I recently gave a talk at a San Francisco event called Nosql, and it was recorded and uploaded to the internet. I provided an overview of where HBase was for the next release (0.20) and talked a little bit about Stumbleupon’s experiences.

NOSQL – HBase from martind on Vimeo.

Recently, Stumbleupon hosted the HBase User Group meeting at our office. This event was followed up by the HBase hackathon where many committers and contributors gathered to plan the work for the next major revision of HBase. Getting together every so often allows us to agree on work plans, in which we then can go back and continue to coordinate via IRC and email lists for bug reports.

In addition to my open source project work, I am busy making HBase power various projects here at Stumbleupon. The first project to launch using HBase as it’s storage backend is our short-url service su.pr. We use HBase to power all the data and real-time analytics. There are more applications of Hadoop and cascading behind the scenes as well.

HBase has a bright future at Stumbleupon. With the release of 0.20, we are pushing forward on the next set of features, and looking to expand our use of HBase. Follow me on twitter at @ryanobjc