[Spark-User] Run spark unit test on Windows 7

Hi all,
I'm trying to run some transformation on *Spark*, it works fine on cluster
(YARN, linux machines). However, when I'm trying to run it on local machine
(*Windows 7*) under unit test, I got errors:
java.io.IOException: Could not locate executable null\bin\winutils.exe
in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
at org.apache.hadoop.util.Shell.(Shell.java:326)
at org.apache.hadoop.util.StringUtils.(StringUtils.java:76)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
My code is following:
@Test
def testETL() = {
    val conf = new SparkConf()
    val sc = new SparkContext("local", "test", conf)
    try {
        val etl = new IxtoolsDailyAgg() // empty constructor
        val data = sc.parallelize(List("in1", "in2", "in3"))
        etl.etl(data) // rdd transformation, no access to SparkContext or Hadoop
        Assert.assertTrue(true)
    } finally {
        if(sc != null)
            sc.stop()
    }
}
Why is it trying to access hadoop at all? and how can I fix it? Thank you
in advance
Thank you,
Konstantin Kudryavtsev

Reply To : Run Spark Unit Test On Windows 7

asked Jul 2 2014 at 09:38

Konstantin Kudryavtsev

7 Replies for : Run Spark Unit Test On Windows 7

Hi Konstatin,
We use hadoop as a library in a few places in Spark. I wonder why the path
includes "null" though.
Could you provide the full stack trace?
Andrew
2014-07-02 9:38 GMT-07:00 Konstantin Kudryavtsev <
[email protected]>:

Reply To : Run Spark Unit Test On Windows 7

answered Jul 2 2014 at 10:15

Andrew Or

Hi Andrew,
it's windows 7 and I doesn't set up any env variables here
The full stack trace:
14/07/02 19:59:31 WARN NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
14/07/02 19:59:31 ERROR Shell: Failed to locate the winutils binary in the
hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in
the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
 at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
at org.apache.hadoop.util.Shell.(Shell.java:326)
 at org.apache.hadoop.util.StringUtils.(StringUtils.java:76)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
 at org.apache.hadoop.security.Groups.(Groups.java:77)
at
org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)
 at
org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)
at
org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:283)
 at org.apache.spark.deploy.SparkHadoopUtil.(SparkHadoopUtil.scala:36)
at
org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala:109)
 at org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala)
at org.apache.spark.SparkContext.(SparkContext.scala:228)
 at org.apache.spark.SparkContext.(SparkContext.scala:97)
at my.example.EtlTest.testETL(IxtoolsDailyAggTest.scala:13)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
 at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
 at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
 at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
 at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
 at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:81)
at org.junit.runner.JUnitCore.run(JUnitCore.java:130)
 at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
at
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
 at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Thank you,
Konstantin Kudryavtsev

Reply To : Run Spark Unit Test On Windows 7

answered Jul 2 2014 at 10:20

Konstantin Kudryavtsev

By any chance do you have HDP 2.1 installed? you may need to install the utils and update the env variables per http://stackoverflow.com/questions/18630019/running-apache-hadoop-2-1-0-on-windows

Reply To : Run Spark Unit Test On Windows 7

answered Jul 2 2014 at 11:10

Denny Lee

No, I don�t
why do I need to have HDP installed? I don�t use Hadoop at all and I�d like to read data from local filesystem

Reply To : Run Spark Unit Test On Windows 7

answered Jul 2 2014 at 12:04

Kostiantyn Kudriavtsev

You don't actually need it per se - its just that some of the Spark
libraries are referencing Hadoop libraries even if they ultimately do not
call them. When I was doing some early builds of Spark on Windows, I
admittedly had Hadoop on Windows running as well and had not run into this
particular issue.

Reply To : Run Spark Unit Test On Windows 7

answered Jul 2 2014 at 12:24

Denny Lee

It sounds really strange...
I guess it is a bug, critical bug and must be fixed... at least some flag
must be add (unable.hadoop)
I found the next workaround :
1) download compiled winutils.exe from
http://social.msdn.microsoft.com/Forums/windowsazure/en-US/28a57efb-082b-424b-8d9e-731b1fe135de/please-read-if-experiencing-job-failures?forum=hdinsight
2) put this file into d:\winutil\bin
3) add in my test: System.setProperty("hadoop.home.dir", "d:\\winutil\\")
after that test runs
Thank you,
Konstantin Kudryavtsev

Reply To : Run Spark Unit Test On Windows 7

answered Jul 2 2014 at 23:44

Konstantin Kudryavtsev

Hi Konstantin,
Could you please create a jira item at: https://issues.apache.org/jira/browse/SPARK/ so this issue can be tracked?
Thanks,
Denny
It sounds really strange...
I guess it is a bug, critical bug and must be fixed... at least some flag must be add (unable.hadoop)
I found the next workaround :
1) download compiled winutils.exe from http://social.msdn.microsoft.com/Forums/windowsazure/en-US/28a57efb-082b-424b-8d9e-731b1fe135de/please-read-if-experiencing-job-failures?forum=hdinsight
2) put this file into d:\winutil\bin
3) add in my test: System.setProperty("hadoop.home.dir", "d:\\winutil\\")
after that test runs
Thank you,
Konstantin Kudryavtsev
You don't actually need it per se - its just that some of the Spark libraries are referencing Hadoop libraries even if they ultimately do not call them. When I was doing some early builds of Spark on Windows, I admittedly had Hadoop on Windows running as well and had not run into this particular issue.
No, I don’t
why do I need to have HDP installed? I don’t use Hadoop at all and I’d like to read data from local filesystem
By any chance do you have HDP 2.1 installed? you may need to install the utils and update the env variables per http://stackoverflow.com/questions/18630019/running-apache-hadoop-2-1-0-on-windows
Hi Andrew,
it's windows 7 and I doesn't set up any env variables here 
The full stack trace:
14/07/02 19:59:31 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/07/02 19:59:31 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
at org.apache.hadoop.util.Shell.(Shell.java:326)
at org.apache.hadoop.util.StringUtils.(StringUtils.java:76)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
at org.apache.hadoop.security.Groups.(Groups.java:77)
at org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)
at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)
at org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:283)
at org.apache.spark.deploy.SparkHadoopUtil.(SparkHadoopUtil.scala:36)
at org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala:109)
at org.apache.spark.deploy.SparkHadoopUtil$.(SparkHadoopUtil.scala)
at org.apache.spark.SparkContext.(SparkContext.scala:228)
at org.apache.spark.SparkContext.(SparkContext.scala:97)
at my.example.EtlTest.testETL(IxtoolsDailyAggTest.scala:13)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:232)
at junit.framework.TestSuite.run(TestSuite.java:227)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:81)
at org.junit.runner.JUnitCore.run(JUnitCore.java:130)
at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:120)
Thank you,
Konstantin Kudryavtsev
Hi Konstatin,
We use hadoop as a library in a few places in Spark. I wonder why the path includes "null" though.
Could you provide the full stack trace?
Andrew
2014-07-02 9:38 GMT-07:00 Konstantin Kudryavtsev :
Hi all,
I'm trying to run some transformation on Spark, it works fine on cluster (YARN, linux machines). However, when I'm trying to run it on local machine (Windows 7) under unit test, I got errors:
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
at org.apache.hadoop.util.Shell.(Shell.java:326)
at org.apache.hadoop.util.StringUtils.(StringUtils.java:76)
at org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
My code is following:
@Test
def testETL() = {
    val conf = new SparkConf()
    val sc = new SparkContext("local", "test", conf)
    try {
        val etl = new IxtoolsDailyAgg() // empty constructor
        val data = sc.parallelize(List("in1", "in2", "in3"))
        etl.etl(data) // rdd transformation, no access to SparkContext or Hadoop
        Assert.assertTrue(true)
    } finally {
        if(sc != null)
            sc.stop()
    }
}
Why is it trying to access hadoop at all? and how can I fix it? Thank you in advance
Thank you,
Konstantin Kudryavtsev

Reply To : Run Spark Unit Test On Windows 7

answered Jul 3 2014 at 09:06

Denny Lee

Related discussions

How To Setup Solr To Run On Production In A Windows 7 Environment

Hi, I have been looking for documentation detailing how to setup/deploy solr to run on our production servers (windows 7 environment with tomcat). I have created a simple search app locally and would like to put the app and the solr index on our test server for others test. Any assistance will be greatly appreciated. Thanks, Jackie

Any Limitation Running Mongo Instances On Windows 7 For Production"

Hi Is there anyone here running multiple mongo instances on windows 7 for production? So far I have tested instances on windows server 2008 r2. But those VMs are msdn license. And from my understanding you new multiple shard and replica instances for load balancing and ha, and ms server licenses could be an issue. If our shop allow linux, it would be great, we don't have enough people

Running Hive On Windows 7

Hi I'm a freshman in hadoop world. After some struggling, i've successfully make hadoop 2.6 running on my windows 7 laptop. However when I want to run hive 1.0.0 on my win 7 system, I found there is no cmd line script as provided for linux. It's also hard to find any useful message in google. That's why i seek here. Anyone can provide me any clue on how to run hive on window 7

Running Hive On Windows 7

Newbie Question On Running System Commands On Windows 7

I’m new to scala and I’m trying to run a system command in a scala script. In Linux, can do: import sys.process._ val result = ("ls").! and it works as expected. In windows: import sys.process._ val result = ("dir").! Gives an error: java.io.IOException: Cannot run program "dir": CreateProcess error=2, The system cannot find the file specified

Pig On Windows 7

Hello all, Is there anyone who have installed Pig on Windows 7, and running successfully? I have an issue with my installation and need help. Thanks, Venkat.

Tomcat 5.5 And Windows 7

Hi there, I only wanna know if i can run Tomcat 5.5 on windows 7 ??? And if i have to take any considerations to do it. Thanks for yours answers. Regards Marcos.

HBase On Windows 7: Master Not Running Error

Hi all, I've been trying to get HBase (version 0.94.4) working on my system with Windows 7 for several days now to no avail. I've followed the instructions at http://hbase.apache.org/cygwin.html and tried to get to the bottom of the problem myself, but it looks like now I'm stuck. The weird thing is also that after installing HBase the first time I can use it without problems, but after stopping

Running Tomcat In Debug Mode On Windows 7

I want to run TomCat in debug mode so I can debug my servlet from Eclipse. I installed TomCat 7 using the Windows Installer. It was installed as a service, managed by the Commons Daemon Service Manager. It runs servlets fine (i.e., no problems with normal TC operation). I did the following: 1. Added "jpda start" as Arguments in the Startup tab in the Daemon Service Manager. 2. Created a custom Debug

Need Help To Run Redis On Windows 7 32 Bit Os

� � � �I'm using�windows 7 32 bit. I download redis 2.8.8 through ur website�http://redis.io/download. When i extract downloaded redis 2.8.8 ZIP file. I did't get redis-server.exe file under redis 2.8.8 DIRECTORY and also redis-cli.exe file on src DIRECTORY � � � I am not able to run redis on my system(Windows 7 32bit) and i got struck from past 3 days..Any help would be appreciable..!! NOTE: I need

Cannot Connect To CouchDb On Windows 7 64 Bit Professional

Hello, Service starts successfully but cannot connect.Tried netstat but don't see the port used. No log file created. Trying to run from Start menu (couchdb.bat) I get this in Erlan console: Erlang R14B04 (erts-5.8.5) [source] [smp:16:16] [rq:16] [async-threads:0] Eshell V5.8.5 (abort with ^G) 1> {"init terminating in do_boot",{{badmatch,{error,{{app_would_not_start,os_mon},{

Can't Install Socket.io Or Run Now.js On Windows 7 X64

Running on a Windows 7 Home Premium 64-bits Downloaded: node.js (0.8.7-x64) as a windows msi from the official site make-3.81 from http://gnuwin32.sourceforge.net/packages/make.htm (Complete package, except sources) Microsoft visual Studio 2012 Express for web And then followed the steps at http://blog.nowjs.com/running-nowjs-natively-on-windows, which implied on: installing the Microsoft Visual

How To Run Mongod On Win 7

I am newbie t mongodb. I have installed Mongodb service on my Win 7 system. The service is running. When I run 'mongod' command, Windows gives me message that this command is not found. Should I set environment variable ? Can some one guide me what Iam doing wrong and how it can be rectified. Thanks in advance

[redis-db] Trying To Run Redis On Windows 7-Unsuccesfully

Hi i am using virtualbox with windows 7 on-board. I am trying to run Redis server (downloaded executable installation file from GitHub). When trying to run Redis client i am getting the following error: Could not connect to Redis at 127.0.0.1:6379: Unknown error (1.00s) not connected> I also cannot run any 'redis' commands in the CLI. What would you suggest to do? You received

Neo4j-enterprise-1.7-SNAPSHOT Can`t Run On Windows 7 64bit ?

Report the Error: Unable to access jarfile E:document eo4jNeo4j NEO4J-~1.7-Sinwindows-service-wrapper-5-SNAPSHOT.jar when I double-click on Neo4j.bat at the $Neo4j-homein?

Problem With Czech Character On Windows 7

Hello, I have this part of code and PDF included special Czech character ---- PDFTextStripperByArea stripper = new PDFTextStripperByArea(); stripper.setSortByPosition( true ); Rectangle rect = new Rectangle ( 10, 280, 275, 60 ); stripper.addRegion( "class1", rect ); List allPages = document.getDocumentCatalog().getAllPages(); PDPage firstPage = (PDPage)allPages.get( 0 );

[[email protected]] Trouble Installing On Windows 7

Hi, I am trying to install Apache HTTP Server v2.2.14 on a machine recently upgraded to Windows 7. Apache worked fine under Windows XP, but when I go to install this in Windows 7, the install script appears to hang. In the install wizard dialog it states: Installing Apache HTTP Server 2.2.14 The program features you selected are being installed. Please wait while the Installation Wizard installs

MySQL On 64 Bit Windows 7?

Although 74 bit Windows 7 is listed as supported [1], I do not see such a binary listed on the download page [2]. Should one use the 32 bit installer on 64 bit Windows? Is the installer page sniffing my UA (Firefox on Debian) and trying to guess as to the correct binary for me? Am I looking in the wrong place? Thanks. [1] http://www.mysql.com/support/supportedplatforms/database.html [

Template Not Rendering On Windows Phone 7

We noticed that templates are not being rendered properly: we see the placeholders instead of the model data. The user agent in this case is: "Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0; LG; LG-E900)" Do we need to add some polyfill script to make IEMobile 9 work?

Windows Installer Trying To Install "Windows Azure Command Line Tools For Mac And Linux" On Windows 7

Hi, When I try to run the Powershell script found in these instructions, the script tries to install "Windows Azure Command Line Tools for Mac and Linux". Of course if fails, as I am installing it from Windows 7. How can I fix this?

Run Spark Unit Test On Windows 7

Related discussions

Spark-dev

Spark-user