Open Source to the Rescue

For those who still don't get IT, an explanation for the evolution of Friendster's open source architecture should help...consider: "There were discussions of ditching open source. Should we just buy off-the-shelf software that would do what we want?" Pattishall explains. "We investigated the market but found there wasn't anything close to what we wanted to do, and it was way too expensive. We wouldn't be able to execute on any business plan because we spent too much just on the software." Read on...

Open Source to the Rescue [1]

By Cathleen Moore - InfoWorld - 04/25/05

In founding Friendster, Chairman Jonathan Abrams sought to create an online network through which friends could connect with friends. When it launched, the service was powered by a Java back end running on Apache Tomcat servers with a MySQL database. That original architecture was soon crushed by the coming load of traffic.

Who says open source can't measure up to commercial software for mission-critical applications? Far from being a mere quick fix or low-cost alternative, open-source software is helping real-world companies solve their most pressing IT problems.

Perhaps no more dramatic example exists than pioneering social networking site Friendster. When Friendster launched in March 2003, no one imagined that within two years the site would reach 60 million page views per day. Unfortunately, as the site's traffic increased, so did its performance issues. The problem, in essence, was that Friendster had unexpectedly become a phenomenon.

"When I arrived it was a crisis point -- absolutely, all day, every day," says Chris Lunt, Friendster's director of engineering, who joined the company in the summer of 2003. At that time, he says, Friendster's architecture was nearly breaking beneath the traffic load.
Fast Track

"[Friendster] had taken off much faster than anyone could anticipate," Lunt says. "We had our millionth user [when] the site had been up only six months. The thing was overwhelmed."

Friendster's performance problems needed to be solved, fast. Rather than stick to the paved road of commercial software, however, the company's engineers took a major risk by betting on the open-source application stack known as LAMP, which consists of and is named for -- the Linux OS, Apache Web server Get the IBM Storage Resource Management Whitepaper. Latest News about Servers, MySQL database, and PHP (PHP: Hypertext Preprocessor) scripting language.

Fortunately, that gamble paid off. LAMP not only allowed Friendster's engineers to scale the site's architecture to address its unwieldy growth, but along the way, they implemented creative configurations that brought the LAMP technologies themselves to a new level.
Brought to Its Knees

In founding Friendster, Chairman Jonathan Abrams sought to create an online network through which friends could connect with friends. When it launched, the service was powered by a Java Latest News about Java back end running on Apache Tomcat servers with a MySQL database. That original architecture was soon crushed by the coming load of traffic.

During the summer of 2003, Friendster was plagued by performance issues. Often, the millions of users pounding the site where unable to access it, and when they could, results were inconsistent from page to page. User profile changes failed to show up because of lags in the distributed architecture, and messages were dropped.

"If you had a huge network [of friends], you couldn't search ePilot Search Engine Advertising - Click here to add your site today. it because just building your list and comparing to the network took longer than the browser would allow you to wait," says Dathan Pattishall, senior database and software engineer at Friendster. Pattishall joined the company in November 2003 to tackle the site's database issues.

Tomcat and Java weren't the problem so much as the fact that the site's back end was not architected to accommodate millions of users. Friendster had grown to such a huge extent that simply throwing more hardware at the problem wasn't enough. The site had to be re-engineered to make better use of the hardware and applications.

Of course, that was easier said than done. At the time, Friendster's IT team consisted of two engineers, and the challenges they faced were daunting.

"Developing for your desktop is one thing, but when [your site needs to support] millions of hits a day, it is a different story," Lunt says.

The Tomcat-based Java implementation was bulky, difficult to manage, and couldn't scale to meet the surge in traffic. Moreover, the MySQL back end was a bottleneck. According to Pattishall, "Every application and subapplication had a problem. And we had to fix it."
Lighting the LAMP

Lunt had some prior job experience in the late 1990s with a variety of open source combinations, including running MySQL and Linux on Intel (Nasdaq: INTC) Latest News about Intel hardware and developing applications using Perl as a scripting language. He had also experimented with PHP, a language similar to Perl but optimized for Web development. After considering his options, Lunt led the charge to completely re-architect Friendster using Apache and PHP.

"We had to go back and rewrite the site," Lunt says. "That was not a popular choice here. They had a couple Tomcat authors here who were pretty threatened by that."

Still, Friendster pushed forward with LAMP because it was the only toolset that would allow the company to create its vision, according to Pattishall. "The one thing we kept the same was using open source software - using MySQL, code we could look at, and stuff we could download off the Net. This let us do things [our own] way," he says.

Pattishall says the benefits of open source include better visibility in the architecture and more control of fixes. "With open source you can look directly at what is causing a problem, write your own fix, and get up and running in such a fast manner. Instead of waiting four weeks for a vendor to provide a qualified patch, you provide the fix yourself," he says.

The transition to LAMP wasn't entirely smooth. "What we were doing was challenging and hard. We were a top 40 Web site Get a Free E-Commerce Start-up Kit from Verisign. We were dealing with a volume of traffic that most sites don't dream of," Lunt says.

Late-night emergencies were common, and firefighting consumed the whole of many days. The problems were so severe that discussions were had as to whether it might be more prudent to go a more traditional engineering route.

"There were discussions of ditching open source. Should we just buy off-the-shelf software that would do what we want?" Pattishall explains. "We investigated the market but found there wasn't anything close to what we wanted to do, and it was way too expensive. We wouldn't be able to execute on any business plan because we spent too much just on the software."

What's more, although taking the open-source route may seem intimidating because of the difficulty in obtaining support, Lunt says traditional vendor support is not always a safety net, adding, "Commercial software support contracts are often a support blanket."
Re-architecting the Data

Some of Friendster's biggest headaches came from the site's MySQL implementation, according to Pattishall. "Everything was wrong with the usage of MySQL when I came in," he says. "Everything was going through one location, which was a huge bottleneck. All our writes were taking forever."

Rather than ditch MySQL for a large commercial RDBMS such as Oracle (Nasdaq: ORCL) Latest News about Oracle, however, Friendster's IT team decided to rebuild its MySQL implementation, this time engineering it the right way.

"We worked to customize MySQL. We had great support from [MySQL AB, the parent company of the software]. We wouldn't be able to do this with Oracle," Lunt says. "The licensing costs would kill us.... It's a good product, but not that much better to justify the cost."

One problem Friendster's engineers had to address was optimizing the way queries were performed against the database. "Imagine asking a question to a database, and in that question, you ask 700,000 to 5 million questions to be answered all at the same time. That was what was being done to produce the social network part of site," Pattishall says.

A key design goal for the new system was to move away from maintaining session state toward a stateless architecture that would clean up after each request, Lunt says. In addition, according to Pattishall, the team rebuilt the MySQL data in a more compact form, re-indexing and repairing tables in order to stabilize the site, improve page load times, and better handle traffic.

Another important step in stabilizing the site was getting more hardware in place and setting up a system to automate the hardware selection process. "Rather than buy big, centralized boxes, [our philosophy] was about buying a lot of thin, cheap boxes. If one fails, you roll over to another box," Lunt says.

Friendster changed to a different white-box vendor, Open Source Storage.

"They gave us rock-solid, stable machines so we could skip time in qualifying the actual hardware itself," Pattishall says.

According to Lunt, another key step to stabilizing the service was to install a monitoring system that could alert engineers when a system or hardware piece failed. For that part of the puzzle, Lunt chose the open source network monitoring program Nagios.
Creative Configurations

With site stability within reach, Pattishall developed some creative MySQL tools of his own that wouldn't have been possible with off-the-shelf technologies. To address the lack of management features available with MySQL, Pattishall developed a management tool that provides a real-time view of a cluster and that graphs database statistics across time. Pattishall is releasing the tool as an open source project.

Another problem was the management of the hardware, which proved labor-intensive; disks often failed, requiring manual replacement. Lunt and Pattishall looked for a better way to manage the systems, eventually creating a unique configuration consisting of 64-bit Get your complimentary copy of the IBM 64-Bit Computing Decision-Maker's Guide. AMD (NYSE: AMD) Latest News about AMD Opteron hardware running on Linux with a custom kernel for talking to a SAN.

"The boxes connect over a fiber link, so if a box fails, we don't lose all those disks and can get back up quickly. No one was using the configuration we're using," Pattishall says.

Pattishall also developed a custom configuration to load balance MySQL traffic using an appliance from NetScaler that was originally designed for accelerating Web pages.

"When our Web server makes a database [request], from the application point of view it looks like it is connecting to a single server. Because of the load balancer, we can take one request and put on 20 boxes. We are able to handle more load than if we stuck all those requests on a single box," Pattishall says.

Development on the new version of the Friendster site running Apache and PHP was completed in June 2004. In the process, Friendster's development team had grown to 20 engineers, primarily focused on rolling out new features to the service.

"Open source makes it easier to do that. The majority of new features have been built on open source," Pattishall says.

According to Tim Denike, senior Unix administrator at Friendster, open source was the only solution to conquer the site's massive growth problems. "Open-source tools allowed us to scale a massively complex application across a system that required very little administrative overhead compared to those of other companies," he says.

So the next time your company has to weigh its technology options for an IT project, don't hesitate to throw open source into the mix. Mature open-source technologies really are ready for prime time; the new and improved Friendster is living proof.