Scribd
Upload
9 views4 pages

codegen deployment project flow

Uploaded by

Anonymous xMYE0TiNBc
Copyright
© © All Rights Reserved
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
Download as txt, pdf, or txt
9 views4 pages

codegen deployment project flow

Uploaded by

Anonymous xMYE0TiNBc
Copyright
© © All Rights Reserved
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
Download as txt, pdf, or txt
Download as txt, pdf, or txt
You are on page 1/ 4

DBT MODELS - TRANSFROMATION LOGIC

DBT SEEDS - CSV FILES


DBT SOURCES -
DBT MACROS
DBT VARIABLES
DBT HOOKS
DBT SNAPSHOTS
DBT INCREMENTAL
DBT EPHEMERAL
DBT TEST
DBT PACKAGES
============================
DBT UTILS
DBT EXPECTATIONS

table :
100 columns
select * from tb;

select col1,col2,col3,col4,col5..

from tb;

{{ codegen.generate_base_model(
source_name='raw_jaffle_shop',
table_name='customers',
materialized='view'
) }}

{{ codegen.generate_source(schema_name= 'raw_sc', database_name=


'dev_db',include_schema=True,
name='prav') }}

===========================
DEVELOPMENT
TESTING - DBT TEST
DBT DOCS GENEARATE
DATA LINAAGE
ELT DATA PIPEKINES ;

DEPOLYMNETS:

MOVING THE DB OBJCTE CFOME ONE EVBIRNINT TO ANITHE NEVIR

TABLE
DEV => TEST =>STG =>PORD

1.
github account creation
2. create 1 reposiirry

3. copy ssh
git@github.com:dbtpraveen3/dbt-sf-scripts.git
put the ssh value in the git clone

copy the ssh value and put in github

ci cd ??

DEV =>STG
DEV =>PROD

ORACLE TO SNOWFLAKR

ORACLE TO S3 ?? TASK

UI DIAGRAM
dev =>prod
=====================DEPOLYMENT
1. CREATE ENVIRONMENT

3 to 4 developers
code reposiory or version control
github

any devlopmemt:
we need to take the code from

laetst code
;

DEVELOPEMNT
ETSTING
PUT THE CODE IN THE REPOSIROY

MONTH QUKAREY O;

DEVOPS OR SUPPORT GUY THEY TRY TO DEPLOT YHE COIE IN DEIFFENT ENFIRNAONNT;

3RD PART ORCHETSTATION OR SCHEDULER TOOLS??

AIRFLOW/PERFECT
CONTROL-M

EXTERNAL DEPENDECCIDES ??
ANY FAILURE HAPPENDING

SLACK OR MAIL
WILL GET NOTIFICATION ?

====================

DBT PROJECT FLOW :

DIFFERNT SOURCE SYSTEM :


ORACEL
SQL SERVER
PISTGRE =>ELT TOOL(FIVETRAN)=> SNOWFLAKE()
SAP HANA

BACTH PROCESS ??

SNOWFLAKE(RAW_SC/RAW_sC1/RAW_SC2 TRANSIENT TABLE)


===================>DBT
SOURCES.YML ??? codegen genatesource
===>

we want bsic data maasaging


rmvoeing the spaces
based on the mapping docuente give proer name

column length

dbt models ??
spefcial charter ?? dbt macros
global variables
__var gunction

raw=>dbt =>stg(modles) =>sanitize(ref fucntoion)

i want to duplcaites
data quality ??DBT TEST(SINGUKAR GENRICE CUSTOM GENEIRC)
INCREMENATL
incrmenatl materilziation
SNApSHOTS
=================

final layer

i want to join multipole models


ref fucntion ??
=================

reporting views

{{config(materialized='view'}};

data sharing :
data unlaod
dbt macris

data sharing :
secure
{{config(materialized='view',secure=true}};

hooks
packages

SATURDAYA :
WORKING
8AM
I DID SOME CHANGE IN THE DEV ??
OCNCE WE COMMITED THE CIDE NEEDS TO AUTOAMTELY RFLECTED IN THE STG

You might also like