Loading

• EVENTS
• Upcoming
• Calendar
• Colloquia
• TCSDLS
• Student Talks
• Research Seminars
• inDuke Seminars
• Search
• Archive
• 2013-2014 2012-2013 2011-2012 2010-2011 2009-2010 2008-2009 2007-2008 2006-2007 2005-2006 2004-2005 2003-2004 2002-2003 2001-2002 2000-2001 1999-2000 1998-1999 1997-1998 1996-1997 1995-1996 1994-1995

Cloud computing provides us with interesting new options for performing statistical analysis tasks on big data. Cloud service providers offer a variety of choices in termsof machine types and cluster sizes in an on-demand, pay-as-you-go fashion. The MapReduce system, with its superior scalability, elasticity, and fault-tolerance features, serves as a nice massively data-parallel computing platform on these cloud-based clusters. In this project, we build an end-to-end system for large-scale statistical computing based on Hadoop MapReduce, with improved storage and execution engines and automatic optimization and provisioning for the cloud setting.

By introducing special support for big matrices at both storage and execution levels, we are able to run statistical workloads much faster by reducing the number of MapReduce jobs required per workflow to about a half compared with previous works. Furthermore, we automatically choose the desired cluster configuration for carrying out a given workload.

Jobs involving different statistical operations (CPU-, network-, or I/O-intensive) prefer different machine types in terms of time and cost. The best job execution plan also changes as the cluster changes. A clever choice of execution plan along with cluster could save us a lot of money. Manual plan selection is hard, thus we built a cost- based optimizer that can solve these optimizations systematically and provide the best execution plan and cluster provisioning plan to the user automatically. With this optimizer, we can answer interesting questions like: “what is the minimum budget required to execute the workflow by a given deadline?”

Advisor(s): Jun Yang & Shivnath Babu

Alvin Lebeck