Programming Resources

This is a good paper to read at  http://labs.google.com/papers/mapreduce.html

Quoted from the abstract

MapReduce is a programming model and an associated implementation for processing and generating large data sets. Users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. Many real world tasks are expressible in this model, as shown
in the paper.

Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. The run-time system takes care of the details of partitioning the input data, scheduling the program’s execution across a set of machines, handling machine failures, and managing the required inter-machine communication. This allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Apache Hadoop Wins Terabyte Sort Benchmark

Quoted

One of Yahoo’s Hadoop clusters sorted 1 terabyte of data in 209 seconds, which beat the previous record of 297 seconds in the annual general purpose (daytona) terabyte sort benchmark. The sort benchmark, which was created in 1998 by Jim Gray, specifies the input data (10 billion 100 byte records), which must be completely sorted and written to disk. This is the first time that either a Java or an open source program has won. Yahoo is both the largest user of Hadoop with 13,000+ nodes running hundreds of thousands of jobs a month and the largest contributor, although non-Yahoo usageand contributions are increasing rapidly.

Hadoop is a software platform that lets one easily write and run applications that process vast amounts of data.

Hadoop implements MapReduce, using the Hadoop Distributed File System (HDFS).

I haven been using Apache Commons Logging in all my projects previously. Recently I was trying SL4J in one of the project, and found that it is a good Java logging facade framework.

As quoted from the website, the Simple Logging Facade for Java or (SLF4J) is intended to serve as a simple facade for various logging APIs allowing to the end-user to plug in the desired implementation at deployment time. SLF4J also supports a bridging legacy APIs as well as a source code migration tool.

SLF4J API offers an advanced abstraction of various logging systems, including JDK 1.4 logging, log4j and logback. Features include parameterized logging and MDC support.

Logging systems can either choose to implement the the SLF4J interfaces directly, e.g. logback or SimpleLogger. Alternatively, it is possible (and rather easy) to write SLF4J adapters for a given API implementation, e.g.Log4jLoggerAdapter or JDK14LoggerAdapter.

For those using JCL, you can give it a try and decide which framework you want to use.

ZoneMinder is a Linux video camera security and surveillance solution. ZoneMinder is intended for use in single or multi-camera video security applications, including commercial or home CCTV, theft prevention and child, family member or home monitoring and other domestic care scenarios such as nanny cam installations. It supports capture, analysis, recording, and monitoring of video data coming from one or more video or network cameras attached to a Linux system. ZoneMinder also support web and semi-automatic control of Pan/Tilt/Zoom cameras using a variety of protocols. It is suitable for use as a DIY home video security system and for commercial or professional video security and surveillance. It can also be integrated into a home automation system via X.10 or other protocols.

ZoneMinder is an integrated set of applications which provide a complete surveillance solution allowing capture, analysis, recording and monitoring of any CCTV or security cameras attached to a Linux based machine. It is designed to run on distributions which support the Video For Linux (V4L) interface and has been tested with video cameras attached to BTTV cards, various USB cameras and IP network cameras. A partial list is given in the Wiki and Support sections. ZoneMinder also requires MySQL and PHP, and is enhanced by a webserver such as Apache.

Another interesting project

The RODS Open Source Project - Open Source Outbreak and Disease Surveillance Software

This is interesting..

You can now upload your own data set and create data visualization with Many Eyes

The heart of Many Eyes is a collection of data visualizations.

On Many Eyes you can:

• View and discuss visualizations
• View and discuss data sets
• Create visualizations from existing data sets
• Rate data sets and visualizations
• Upload your own data
• Create and participate in topic hubs
• Select items to watch
• Track your contributions, watchlist, and topic hubs
• See comments that others have written to you

SafeSpace eliminates the problem of malware by creating a secure barrier between your computer and the internet, protecting you and your personal information against known and unknown malware threats. SafeSpace isolates internet facing applications in a virtual sandbox, which protects your system files and settings from unwanted changes, and blocks unauthorized access to your private files and data. When you next log off, or purge SafeSpace all virtualized changes are wiped, leaving your computer clean and malware free. Downloaded files are tracked by SafeSpace to ensure that they always open in the virtual sandbox. Tracked files can be saved to any disk location and stored alongside private files without any additional management. Surf in complete safety. Download with absolute confidence. Log off knowing your computer is clean and your personal information remains private.

Here are 2 articles that you should read if you are using Eclipse with Mylyn

Mylyn 2.0, Part 1: Integrated task management

Mylyn 2.0, Part 2: Automated context management

Quoted from the article,

Mylyn (formerly called Mylar) enhances productivity by seamlessly integrating tasks into Eclipse and automatically managing the context of those tasks as you work. Mylyn Project Lead Mik Kersten has updated his two-part guide to using Mylyn. Part 1 introduces Mylyn’s task management facilities and integration with repositories such as Bugzilla. This second half explains how Mylyn’s context management facilities make multitasking easy and reduce information overload when you’re working on large applications in Eclipse.

qooxdoo is a comprehensive and innovative Ajax application framework. Leveraging object-oriented JavaScript allows developers to build impressive cross-browser applications. No HTML, CSS nor DOM knowledge is needed.

It includes a platform-independent development tool chain, a state-of-the-art GUI toolkit and an advanced client-server communication layer. It is open source under an LGPL/EPL dual license.

The RAP project enables developers to build rich, Ajax-enabled Web applications by using the Eclipse development model, plug-ins with the well known Eclipse workbench extenstion points, JFace, and a widget toolkit with SWT API (using qooxdoo for the client-side presentation).

It allows developers to create Rich Internet Applications (RIA) using the OSGi standard component model. The main advantage of this technology is the ability to share the same Java code base for RCP and RIA applications.

You may also want to look at Equinox, which is an implementation of the OSGi R4 core framework specification, a set of bundles that implement various optional OSGi services and other infrastructure for running OSGi-based systems.

LiquiBase is an open source (LGPL), DBMS-independent library for tracking, managing and applying database changes. It is built on a simple premise: All database changes (structure and data) are stored in an XML-based descriptive manner and checked into source control.

LiquiBase aims to provide a solution that:

Supports merging of changes from multiple developers

• Works well with code branches

• A Database Refactoring IDE-Plugin (short demo)

• Isn’t DBMS-specific

• Allows you to manage production data as well as various test datasets

• Works in an environment where multiple machines will be attempting to migrate the database concurrently

• Supports automated change-applying or generation of SQL scripts that can be approved and applied by a DBA

• Supports rolling back changes

• Can create database ”diff“s and can generate change logs from them

• Can auto-generate database change documentation

A good article is available here.

Panotipcode is a project dedicated to making code metrics so widely understood, valuable, and simple that their use becomes ubiquitous, thus raising the quality of software across the industry. It provides a set of open source tools for gathering, correlating, and displaying code metrics.

It packages

• Emma - Unit test code coverage. By changing one line in your build file this can be switched to Cobertura.
• CheckStyle - Validates that your code follows Sun’s standards for Java. If you want to use a different set of rules you only need to change one parameter in your build file.
• JDepend - Sophisticated OO quality metrics and package dependency checking.
• JavaNCSS - Cyclomatic Complexity and size (NCSS) metrics.
• Volatility - Measure change within your projects. Currently this only works with Subversion, but support for other SCM repositories is planned.
• Duplicate Code - Using Simian.
• Panopticode Aggregator - Generates an XML file that integrates ALL of the information gathered above
• Panopticode Reports
  • Powerful visualizations, such as TreeMaps, that allow you to see the overall picture and an amazing amount of detail in a single view.
  • Metric Correlation
  • Historic Data

OpenSolaris is another alternative for us besides Linux based operating systems.

OpenSolaris is an operating system that provides a rich, coherent platform for building and running applications. The majority of the OpenSolaris code base is distributed under an OSI-approved open source license. Code based on existing open source projects continues to be available under its current license(s), and some binary-only components are covered under the OpenSolaris Binary License.

OpenSolaris supports systems that use SPARC® and x86 families of processor architectures: UltraSPARC®, SPARC64, AMD64, Pentium, and Xeon EM64T.

And, Google is also testing on it.

Bakefile is cross-platform, cross-compiler native makefiles generator. It takes compiler-independent description of build tasks as input and generates native makefile (autoconf’s Makefile.in, Visual C++ project, bcc makefile etc.).

Bakefile’s task is to generate native makefiles, so that people can keep using their favorite tools. There are other cross-platform make solutions, but they either aren’t native and require the user to use unfamiliar tools (Boost.Build) or they are too limited (qmake).

WebKit is an open source web browser engine. WebKit is also the name of the Mac OS X system framework version of the engine that’s used by Safari, Dashboard, Mail, and many other OS X applications. WebKit’s HTML and JavaScript code began as a branch of the KHTML and KJS libraries from KDE.

Other interesting related projects

Konqueror

Web Browser for S60

S60WebKit

K Desktop Environment

Acid2 is a test page published and promoted by the Web Standards Project to identify web page rendering flaws in web browsers and other applications that render HTML.

Acid2 tests aspects of HTML markup, CSS styling, PNG images, and data URIs. It should render correctly on any application that follows the World Wide Web Consortium and Internet Engineering Task Force specifications for these technologies. The idea is that if both web sites and web browsers follow agreed-upon industry standards, then any web site will work the same in any web browser.

Programming isn’t just for adults any longer. Young people are learning programming languages from the earliest ages and up.

There are tools and web sites at http://java.sun.com/new2java/learning/young_developers.jsp that teach young people how to program using the Java programming language, as well as languages developed for ease of use.

The site caters for those with

• No programming experience
• A little experience
• Some experience
• Eager to learn more

Interesting site. For those beginner programmer, worth visiting the site..

PHP Simple DOM Parser

• A HTML DOM parser written in PHP5+ let you manipulate HTML in a very easy way!
• Require PHP 5+.
• Supports invalid HTML.
• Find tags on an HTML page with selectors just like jQuery.
• Extract contents from HTML in a single line.

Here are some popular Unix utilities on Windows

1. Cygwin is a Linux-like environment for Windows. It consists of two parts:
• A DLL (cygwin1.dll) which acts as a Linux API emulation layer providing substantial Linux API functionality.
• A collection of tools which provide Linux look and feel.
2. GnuWin32 provides ports of tools with a GNU or similar open source license, to MS-Windows (Microsoft Windows 95 / 98 / ME / NT / 2000 / XP / 2003 / Vista.
3. GNU Utilities for Win32

For the code snippet

   1: System.out.println(StringUtils.isNumeric(“10.0″));

Using Apache Commons Lang, the above will return false. This is one bug I recently found in one of the existing application.

As quoted from the documentation, StringUtils.isNumeric

Checks if the String contains only unicode digits. A decimal point is not a unicode digit and returns false.

To return the correct result, you should use NumberUtils.isNumber

Gears is an open source project that enables more powerful web applications, by adding new features to your web browser:

• Let web applications interact naturally with your desktop
• Store data locally in a fully-searchable database
• Run JavaScript in the background to improve performance