Developing MapReduce

I use eclipse. Is this http://wiki.apache.org/hadoop/EclipsePlugIn
still the best way to develop mapreduce programs in hadoop? Just want
to make sure before I go down this path.
Or should I just add hadoop jars in my classpath of eclipse and create
my own MapReduce programs.
Thanks

When you download the hadoop in its dist(i don't remember  exact name) there is a related plugin.
Go and get it from there.

Hi Mohit
        I'm really not sure how many of the map reduce developers use the map reduce eclipse
plugin. AFAIK majority don't. As Jignesh mentioned you can get it from the hadoop distribution
folder as soon as you unzip the same. 
My suggested approach would be,If you are on Windows OS, you can test run your map reduce
code in two ways.
-set up cygwin in Windows, atop you can set up hadoop and related tools. It is a little messy.
-Use a linux VM image. I'd recommend  Cloudera test VM,as it comes pre configured with the
whole hadoop technology stack. It really segregates the developer from the hassles of installing
the hadoop tools and making them up and running.
In Linux or Mac you can just add the hadoop jars to your class path and run the driver class
as just how you run a java class within eclipse.(Here hadoop would be on standalone mode).
Hope it helps!...
-

Hello, What do I wrong? I have setup two systems to setup a MapReduce Nutch Crawler but the jobs run always on the same "localhost" when I take a view at http://server:7845 seeds/urls is an url list with 500.000 urls # put seed directory in ndfs bin/nutch ndfs -put seeds seeds # crawl a bit bin/nutch crawl seeds -depth 10 I had mapred.map.tasks on 2 and on 4 and 8 I had mapred.reduce.tasks
Hi all, coming from SQL I still have problems in getting a firm understanding on how to use map/reduce functions properly. Are there any good tutorials out there to help? Thanks, Stephan
I am going to switch from Nutch 0.6 to the latest version of Nutch and have a few questions regarding it. - Does map reduce have any benefits in comparison with the old version if I just have a single server? - After I run the crawler it creates segments and index, but it also creates lots of files under the D:\tmp\nutch\mapred directory (I am using Windows). Does Nutch deletes these files
Hello, I am trying to write a MapReduce job for a boolean preference model. The book exmple in 3.6 listing in MiA only supports FileDataModel. That won't work for the MapReduce since the the files will be sent to the job itself and key/value will replace this data model. I played some with the APIs but I didn't get far. Is there a way to get around that? I would appreciate any concrete examples
Hi, When we run a MapReduce job, the logs are stored on all the tasktracker nodes. Is there an easy way to agregate all those logs together and see them in a single place instead of going to the tasks one by one and open the file? Thanks, JM
Dear all, I wanted to ask a question.. Do Hbase Aggregate Functions such as rowcount, getMax, get Average use MapReduce to execute those functions? Thanks :D
Hi, I wanted to know about usual read patterns from MapReduce applications into HDFS based on users' experiences here. I think large requests are more common (32K and above) but wanted to know if small reads (512bytes - 1K) are common too ? Thanks.
FYI Amazons new Hadoop offering: http://aws.amazon.com/elasticmapreduce/ And Cascading 1.0 supports it: http://www.cascading.org/2009/04/amazon-elastic-mapreduce.html cheers, ckw -- Chris K Wensel [email protected] http://www.cascading.org/ http://www.scaleunlimited.com/
Just want to check how many are using AWS mapreduce and understand the pros and cons of Amazon's MapReduce machines? Is it true that these map reduce machines are really reading and writing from S3 instead of local disks? Has anyone found issues with Amazon MapReduce and how does it compare with using MapReduce on local attached disks compared to using S3.
Hi, is there some way to export/store the compiled MapReduce Job in Hive? Thanks a lot JS