A Hue Spark application was recently created. It lets users execute and monitor Spark jobs directly from their browser and be more productive.
We previously released the app with an Oozie submission backend but switched to the Spark Job Server (SPARK-818) contributed by Ooyala and Evan’s team at the last Spark Summit. This new server will enable a real interactivity with Spark and is closer to the community.
We hope to work with the community and have support for Python, Java, direct script submission without compiling/uploading and other improvements in the future!
As usual feel free to comment on the hue-user list or @gethue! About questions directly related to Job Server, participate on the pull request, SPARK-818 or the Spark user list!
Currently only Scala jobs are supported and programs need to implement this trait and be packaged into a jar. Here is a WordCount example. To learn more about Spark Job Server, check its README.
We assume you have Scala installed on your system.
Currently on github on this branch:
git clone https://github.com/ooyala/incubator-spark.git spark-server cd spark-server git checkout -b jobserver-preview-2013-12 origin/jobserver-preview-2013-12
Then type:
sbt/sbt project jobserver re-start
Currently only on github (will be in CDH5b2):
https://github.com/cloudera/hue#getting-started
If Hue and Spark Job Server are not on the same machine update the hue.ini property in desktop/conf/pseudo-distributed.ini:
[spark] # URL of the Spark Job Server. server_url=http://localhost:8090/
Then follow this walk-through and create the example jar that is used in the video demo.
