Scribd
Upload
68 views9 pages

Setup Single Node Cluster

The document discusses three modes for setting up a Hive metastore: embedded Derby metastore, local Postgres metastore, and remote Postgres metastore on a different machine. It also provides steps for setting up the Spark web UI and application history server.

Uploaded by

praja
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
68 views9 pages

Setup Single Node Cluster

The document discusses three modes for setting up a Hive metastore: embedded Derby metastore, local Postgres metastore, and remote Postgres metastore on a different machine. It also provides steps for setting up the Spark web UI and application history server.

Uploaded by

praja
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd

Single Node Cluster

Set up

Copyright 2021@[Link] - All Rights Reserved 1


Subscribe at [Link]
[Link]
Sibaram Kumar
Sr Data Engineer | Spark | Big Data
Linkedin: in/SibaramKumar/
Copyright 2021@[Link] - All Rights Reserved
Subscribe at [Link] 3
3 Modes to Set up
Hive Metastore

Copyright 2021@[Link] - All Rights Reserved 4


Subscribe at [Link]
Mode #1

Embedded Derby Metastore :

Localhost

Hive Service JVM

Drive Metastore Derby

Copyright 2021@[Link] - All Rights Reserved 5


Subscribe at [Link]
Local Metastore : Mode # 2

Localhost

Hive Service JVM Postgress Service JVM

Drive Metastore

Postgress
Drive Metastore
//localhost:6432/metastore

Copyright 2021@[Link] - All Rights Reserved 6


Subscribe at [Link]
Remote Metastore: Mode #3

Localhost Different Machine([Link])

Hive Service JVM

Metastore
Drive
Server JVM
Postgress
Metastore
Drive
Server JVM
/432/metastore //[Link]:6432/metastore

Copyright 2021@[Link] - All Rights Reserved 7


Subscribe at [Link]
Localhost

Hive
Metastore

PostgreSQL
Image

PostgreSQL Client

Copyright 2021@[Link] - All Rights Reserved 8


Subscribe at [Link]
Set up Web UI:
1. Run a job using spark-submit Command.
2. Get the Port when the applications are submitted. <8088>
3. Set up the Firewall to allow port 8088 at our instance.
4. Open a Browser and test. <PublicIP>:8088
5. At this time, we are still unable to access the Application History.
6. Go to History URL and check the Port for Application History. <18080>
7. Also Initiate the Service : /opt/spark3/sbin/[Link]
8. Allow port 18080 in the Firewall at our instance.
9. Access the Application History. <PublicIP:18080>

Copyright 2021@[Link] - All Rights Reserved 9


Subscribe at [Link]

You might also like