程式競賽工作小組你好:
我們在設置多台hadoop環境的時候遇到了瓶頸,無法執行Hadoop程式,希望可以協助我們解決問題。以下是我們設置環境的步驟:
1.從blog下載的映像檔,安裝JDK,並複製成3份。
2.vim /etc/sysconfig/network,修改其中的HOSTNAME = hadoop1,然後reboot。
3.切換到root權限,vim /etc/hosts,修改內容如下:
127.0.0.1 localhost localhost.localdomain localhost
192.168.42.134 hadoop1 hadoop1.localdomain hadoop1
192.168.42.135 hadoop2 hadoop2.localdomain hadoop2
192.168.42.136 hadoop3 hadoop3.localdomain hadoop3
::1 localhost6.localdomain6 localhost6
※IP 皆有對應到一台機器
4.輸入指令
[hadoop@hadoop1 ~]$ ssh-keygen -t rsa -P '' -f ~/.ssh/ida_rsa
...
[hadoop@hadoop1 ~]$ cat ~/.ssh/ida_rsa.pub >> ~/.ssh/authorized_keys
[hadoop@hadoop1 ~]$ chmod 600 ~/.ssh/authorized_keys
[hadoop@hadoop1 ~]$ scp -r ~/.ssh hadoop2:~/
...
[hadoop@hadoop1 ~]$ scp -r ~/.ssh hadoop3:~/
...
[hadoop@hadoop1 ~]$ ssh hadoop2
[hadoop@hadoop2 ~]$ ssh hadoop3
[hadoop@hadoop3 ~]$
5.修改hadoop-site.xml,Namenode及JobTracker都是hadoop1。修改內容如下:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://hadoop1's ip:9000</value>
</property>
</configuration>
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>hadoop1's ip:9001</value>
</property>
</configuration>
6. vim conf/masters,將檔案中的localhost刪除。
7.vim conf/slaves,修改內容如下:
hadoop1
hadoop2
hadoop3
8.將conf目錄下所有檔案copy到其他機器上。
scp –r ~/conf hadoop2:~/
scp –r ~/conf hadoop3:~/
9.確認主機在/etc/hosts 上有正確的對應,JRE/JDK路徑相同。
10.執行 /bin/start-all.sh,顯示如下圖
輸入yes,出現錯誤訊息
[Fatal Error] hadoop-site.xml:13:2: The markup in the document following the root element must be well-formed.
09/07/02 23:09:59 FATAL conf.Configuration: error parsing conf file: org.xml.sax.SAXParseException: The markup in the document following the root element must be well-formed.
Exception in thread "main" java.lang.RuntimeException: org.xml.sax.SAXParseException: The markup in the document following the root element must be well-formed. at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:948)
at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:851)
at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:819)
at org.apache.hadoop.conf.Configuration.set(Configuration.java:301)
at org.apache.hadoop.conf.Configuration.setBoolean(Configuration.java:461)
at org.apache.hadoop.util.GenericOptionsParser.processGeneralOptions(GenericOptionsParser.java:250)
at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:315)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:134)
at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:119)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:59)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:1852)
Caused by: org.xml.sax.SAXParseException: The markup in the document following the root element must be well-formed.
at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(Unknown Source)
at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.xml.parsers.DocumentBuilder.parse(Unknown Source)
at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:879)
... 11 more
宣告失敗!!