Scribd
Upload
13 views

Data Engineering

Uploaded by

Jk nayak
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
13 views

Data Engineering

Uploaded by

Jk nayak
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 3

Data engineering

Data engineering build a system which collects


 Data from various source like RDBMS, EXCEL,
CSV,PDF ETC.. AND STORE IN A WAREHOUSE
FOR PREDICTIVE ANALYSIS , DATA
SCIENCE ,MACHINE LEARNING & AI
 DATA CAN BE IN THE FORM OF STRUCTURE ,
SEMISTRUCTURE, UNSTURCTURE.
 EARLIER TIME DATA WAS GENERATED VERY
LESS BUT NOW HUGE VOLUMES OF DATA IS
GENERATING , HOW TO USE THIS DATA AND
MAKE KEY DESCISSION FOR THEIR BUSINESS
DEVELOPMENT.
 DATA ENGINEERS ROLE IS TO MAKE DATA
PIPELINE FROM VARIOUS SOURCE TO STORE
DATA IN A DATA WAREHOUSE TO MAKE A
PROPER DECISSION MAKING BY DATA
SCIENTIST AND BUSINESS ANALYST.
SKILLS FOR DATA ENGINEERS
1. SQL (FULL KNOWLEDGE)
2.PYTHON (BASIC KNOWLEDGE DATA TYPE,
LOOPS STATEMENT, LIST, TUPLE,
DCITIONARY,CLASSES FUNCTION ETC.. )
3.DISTRIBUTED COMPUTING FRAMEWORK
LIKE HADOOP BASIC (SLOW PERFORMANCE )AND
SPARK ARCHTECTURE(FAST PERFORMANCE) AND
ITS USES.
4.PYSPARK(PYTHON + SPARK)
5.CLOUD TECHNOLOGY (AWS, AZURE, GCP, OCI)
AZURE SERVICES:-
i) AZURE DATA FACTORY :-IT IS DATA
ORCHESTRATION TOOL WHICH IS USED FOR
ETL OR ELT PROCESS. IT IS TOTALLY NO
CODING TOOL ONLY DRAG AN DROP
OPTIONS
ii) AZURE DATA BRICKS:- DATABRICKS IS A
CLOUD VERSION OF SPARK(DATA BRICKS
AND MICROSOFT JOINTLY STARTED)
HERE WE CAN WRITE A PYTHON ,
R,SCALA ,JAVA CODE

iii)AZURE SYNAPSE ANALYTICS FOR DATA


WAREHOUSE

DATABASE

(OLTP SYSTEM)

EXCEL
ADF(AZURE DATA ADB (AZURE DATAWARE
FACTORY COLLECT DATABRICKS HOUSE(SQL DB) BI
DATA FROM DIFFERENT PROCESS
SOURCE
BIGDATA )
CSV
DATA
INTEGRATION(ETL / ELT
)

IOT/ INTERNET ADLS


GEN2(STORAGE)

DATA ENGINEER DATA ANALYST

ANY QUESTON?

You might also like