Skip to content
This repository was archived by the owner on May 11, 2023. It is now read-only.

Commit 6b6f82c

Browse files
davidmyttonccasher
davidmytton
authored and
ccasher
committed
[579] Implement calculation of embodied emissions for AWS
Towards cloud-carbon-footprint/cloud-carbon-footprint#579
1 parent d571a3b commit 6b6f82c

File tree

3 files changed

+236
-12
lines changed

3 files changed

+236
-12
lines changed

Diff for: ‎coefficients.ipynb

+210-12
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,131 @@
3333
"import warnings; warnings.simplefilter('ignore')"
3434
]
3535
},
36+
{
37+
"cell_type": "markdown",
38+
"metadata": {},
39+
"source": [
40+
"## Embodied emissions\n",
41+
"\n",
42+
"### Constants\n",
43+
"\n",
44+
"These constants are used as part of the calculations for the embodied emissions\n",
45+
"factors for each instance type. They're based on [the work done by\n",
46+
"Teads](https://medium.com/teads-engineering/building-an-aws-ec2-carbon-emissions-dataset-3f0fd76c98ac)\n",
47+
"and extracted from [the source Google\n",
48+
"Sheet](https://docs.google.com/spreadsheets/d/1YhtGO_UU9Hc162m7eQKYFQOnV4_yEK5_lgHYfl02JPE/edit#gid=2090774556)."
49+
]
50+
},
51+
{
52+
"cell_type": "code",
53+
"execution_count": null,
54+
"metadata": {},
55+
"outputs": [],
56+
"source": [
57+
"# Manufacturing emissions for a mono socket, low DRAM, no local storage \n",
58+
"# commodity rack server\n",
59+
"BASE_MANUFACTURING_EMISSIONS = 1000 # kgCO2eq\n",
60+
"\n",
61+
"# Commodity rack server lifespan\n",
62+
"RACK_SERVER_LIFESPAN = 4 # years\n",
63+
"\n",
64+
"# Hourly manufacturing emissions conversion factor - linearly amortised\n",
65+
"MANUFACTURING_EMISSIONS = BASE_MANUFACTURING_EMISSIONS / RACK_SERVER_LIFESPAN / 12 / 30 / 24 # gCO2eq/hour\n",
66+
"\n",
67+
"# DRAM Threshold to unlock additional Scope 3 emissions\n",
68+
"DRAM_THRESHOLD = 16 # GB\n",
69+
"\n",
70+
"# Manufacturing emissions for the threshold DRAM amount\n",
71+
"# Based on Dell PowerEdge R740 Life-Cycle Assessment\n",
72+
"# https://docs.google.com/spreadsheets/d/1YhtGO_UU9Hc162m7eQKYFQOnV4_yEK5_lgHYfl02JPE/edit#gid=954946016\n",
73+
"# = 533 kgCO₂eq for 12*32GB DIMMs Memory (384 GB).\n",
74+
"DRAM_MANUFACTURING_EMISSIONS = (533 / 384) * DRAM_THRESHOLD\n",
75+
"\n",
76+
"# Manufacturing emissions per additional CPU\n",
77+
"CPU_MANUFACTURING_EMISSIONS = 100 # kgCO2eq\n",
78+
"\n",
79+
"# Manufacturing emissions per additional HDD\n",
80+
"HDD_MANUFACTURING_EMISSIONS = 50 # kgCO2eq\n",
81+
"\n",
82+
"# Manufacturing emissions per additional SSD\n",
83+
"SSD_MANUFACTURING_EMISSIONS = 100 # kgCO2eq\n",
84+
"\n",
85+
"# Manufacturing emissions per additional GPU Card\n",
86+
"GPU_MANUFACTURING_EMISSIONS = 150 # kgCO2eq\n"
87+
]
88+
},
89+
{
90+
"cell_type": "markdown",
91+
"metadata": {},
92+
"source": [
93+
"### Calculations\n",
94+
"\n",
95+
"Embodied emissions are based on a representative baeline\n",
96+
"(`BASE_MANUFACTURING_EMISSIONS`) with additional factor added for extra\n",
97+
"components - memory, storage, CPUs and GPUs."
98+
]
99+
},
100+
{
101+
"cell_type": "code",
102+
"execution_count": null,
103+
"metadata": {},
104+
"outputs": [],
105+
"source": [
106+
"aws_cpus = pd.read_csv(f'data/aws-instance-cpus.csv')\n",
107+
"\n",
108+
"def calculate_additional_memory_emissions(platform_memory):\n",
109+
" \"\"\"If the platform memory is greater than the baseline, calculate the \n",
110+
" additional emissions.\"\"\"\n",
111+
"\n",
112+
" if float(platform_memory) > DRAM_THRESHOLD:\n",
113+
" #print(instance)\n",
114+
" additional_emissions = float((float(platform_memory) - DRAM_THRESHOLD) * (DRAM_MANUFACTURING_EMISSIONS / DRAM_THRESHOLD))\n",
115+
" \n",
116+
" else: \n",
117+
" additional_emissions = 0.0\n",
118+
"\n",
119+
" return additional_emissions\n",
120+
"\n",
121+
"def calculate_additional_storage_emissions(storage_type, drive_quantity):\n",
122+
" \"\"\"Calculate additional emissions for storage, depending on the storage \n",
123+
" type.\"\"\"\n",
124+
"\n",
125+
" if drive_quantity <= 0:\n",
126+
" return 0.0\n",
127+
"\n",
128+
" if storage_type.lower() == 'ssd':\n",
129+
" factor = SSD_MANUFACTURING_EMISSIONS\n",
130+
" else:\n",
131+
" factor = HDD_MANUFACTURING_EMISSIONS\n",
132+
"\n",
133+
" return float(drive_quantity * factor)\n",
134+
"\n",
135+
"def calculate_additional_cpu_emissions(platform_name, cpu_name):\n",
136+
" \"\"\"Calculate emissions for additional CPUs for the specified cloud\n",
137+
" platform.\"\"\"\n",
138+
"\n",
139+
" if platform_name == 'aws':\n",
140+
" cpus = aws_cpus\n",
141+
" else:\n",
142+
" return 0.\n",
143+
"\n",
144+
" cpu = cpus.query(f'`CPU Name` == \\\"{cpu_name}\\\"')\n",
145+
"\n",
146+
" if int(cpu['Platform Number of CPU Socket(s)']) > 0:\n",
147+
" return float((int(cpu['Platform Number of CPU Socket(s)']) - 1) * CPU_MANUFACTURING_EMISSIONS)\n",
148+
" else:\n",
149+
" return 0.0\n",
150+
"\n",
151+
"def calculate_additional_gpu_emissions(gpu_quantity):\n",
152+
" \"\"\"Calculate additional emissions for any GPUs.\"\"\"\n",
153+
"\n",
154+
" if gpu_quantity > 0:\n",
155+
" return float(gpu_quantity * GPU_MANUFACTURING_EMISSIONS)\n",
156+
" else:\n",
157+
" return 0.0\n",
158+
" "
159+
]
160+
},
36161
{
37162
"cell_type": "markdown",
38163
"metadata": {},
@@ -48,8 +173,9 @@
48173
"metadata": {},
49174
"outputs": [],
50175
"source": [
51-
"# Loads a CSV file then returns each row appended to an array\n",
52-
"def load_append_data(file_name):\n",
176+
"def load_append_list(file_name):\n",
177+
" \"\"\"Loads a CSV file then returns each row appended to a list.\"\"\"\n",
178+
"\n",
53179
" with open(f'data/{file_name}', 'r') as csvfile:\n",
54180
" reader = csv.reader(csvfile)\n",
55181
"\n",
@@ -59,25 +185,25 @@
59185
" \n",
60186
" return data\n",
61187
"\n",
62-
"cpus_amd_epyc_gen1 = load_append_data('amd-epyc-gen1.csv')\n",
188+
"cpus_amd_epyc_gen1 = load_append_list('amd-epyc-gen1.csv')\n",
63189
"assert 'EPYC 7601' in cpus_amd_epyc_gen1\n",
64-
"cpus_amd_epyc_gen2 = load_append_data('amd-epyc-gen2.csv')\n",
190+
"cpus_amd_epyc_gen2 = load_append_list('amd-epyc-gen2.csv')\n",
65191
"assert 'EPYC 7742' in cpus_amd_epyc_gen2\n",
66-
"cpus_amd_epyc_gen3 = load_append_data('amd-epyc-gen3.csv')\n",
192+
"cpus_amd_epyc_gen3 = load_append_list('amd-epyc-gen3.csv')\n",
67193
"assert 'EPYC 75F3' in cpus_amd_epyc_gen3\n",
68-
"cpus_intel_sandybridge = load_append_data('intel-sandybridge.csv')\n",
194+
"cpus_intel_sandybridge = load_append_list('intel-sandybridge.csv')\n",
69195
"assert 'E5-4610' in cpus_intel_sandybridge\n",
70-
"cpus_intel_ivybridge = load_append_data('intel-ivybridge.csv')\n",
196+
"cpus_intel_ivybridge = load_append_list('intel-ivybridge.csv')\n",
71197
"assert 'E5-2609 v2' in cpus_intel_ivybridge\n",
72-
"cpus_intel_haswell = load_append_data('intel-haswell.csv')\n",
198+
"cpus_intel_haswell = load_append_list('intel-haswell.csv')\n",
73199
"assert 'E5-2630 v3' in cpus_intel_haswell\n",
74-
"cpus_intel_broadwell = load_append_data('intel-broadwell.csv')\n",
200+
"cpus_intel_broadwell = load_append_list('intel-broadwell.csv')\n",
75201
"assert 'E5-2683 v4' in cpus_intel_broadwell\n",
76-
"cpus_intel_skylake = load_append_data('intel-skylake.csv')\n",
202+
"cpus_intel_skylake = load_append_list('intel-skylake.csv')\n",
77203
"assert 'Platinum 8160T' in cpus_intel_skylake\n",
78-
"cpus_intel_cascadelake = load_append_data('intel-cascadelake.csv')\n",
204+
"cpus_intel_cascadelake = load_append_list('intel-cascadelake.csv')\n",
79205
"assert 'Gold 6230R' in cpus_intel_cascadelake\n",
80-
"cpus_intel_coffeelake = load_append_data('intel-coffeelake.csv')\n",
206+
"cpus_intel_coffeelake = load_append_list('intel-coffeelake.csv')\n",
81207
"assert 'E-2246G' in cpus_intel_coffeelake"
82208
]
83209
},
@@ -563,6 +689,8 @@
563689
"source": [
564690
"## Azure\n",
565691
"\n",
692+
"### Use stage coefficients\n",
693+
"\n",
566694
"These values go in [`packages/azure/src/domain/AzureFootprintEstimationConstants.ts`](https://github.com/cloud-carbon-footprint/cloud-carbon-footprint/blob/trunk/packages/azure/src/domain/AzureFootprintEstimationConstants.ts)"
567695
]
568696
},
@@ -781,6 +909,8 @@
781909
"source": [
782910
"## AWS\n",
783911
"\n",
912+
"### Use stage coefficients\n",
913+
"\n",
784914
"These values go in [`packages/aws/src/domain/AwsFootprintEstimationConstants.ts`](https://github.com/cloud-carbon-footprint/cloud-carbon-footprint/blob/trunk/packages/aws/src/domain/AwsFootprintEstimationConstants.ts)"
785915
]
786916
},
@@ -954,12 +1084,80 @@
9541084
"#assert float('{:,.2f}'.format(aws_coefficients[\"GB/Chip\"].mean())) == 80.69"
9551085
]
9561086
},
1087+
{
1088+
"cell_type": "markdown",
1089+
"metadata": {},
1090+
"source": [
1091+
"### Embodied emissions"
1092+
]
1093+
},
1094+
{
1095+
"cell_type": "code",
1096+
"execution_count": null,
1097+
"metadata": {},
1098+
"outputs": [],
1099+
"source": [
1100+
"aws_instances_embodied = []\n",
1101+
"\n",
1102+
"for key, instance in aws_instances.iterrows():\n",
1103+
" # Call our calculation methods for each of the additional components\n",
1104+
" additional_memory = calculate_additional_memory_emissions(\n",
1105+
" instance['Platform Memory (in GB)'])\n",
1106+
" additional_storage = calculate_additional_storage_emissions(\n",
1107+
" instance['Storage Type'],\n",
1108+
" instance['Platform Storage Drive Quantity']\n",
1109+
" )\n",
1110+
" additional_cpus = calculate_additional_cpu_emissions(\n",
1111+
" 'aws',\n",
1112+
" instance['Platform CPU Name']\n",
1113+
" )\n",
1114+
" additional_gpus = calculate_additional_gpu_emissions(\n",
1115+
" instance['Platform GPU Quantity']\n",
1116+
" )\n",
1117+
"\n",
1118+
" # Build a dictionary of the instance emissions\n",
1119+
" aws_instances_embodied.append({\n",
1120+
" 'type': instance['Instance type'],\n",
1121+
" 'additional_memory': round(additional_memory, 2),\n",
1122+
" 'additional_storage': round(additional_storage, 2),\n",
1123+
" 'additional_cpus': round(additional_cpus, 2),\n",
1124+
" 'additional_gpus': round(additional_gpus, 2),\n",
1125+
" 'total': round(BASE_MANUFACTURING_EMISSIONS + additional_memory + additional_storage + additional_cpus + additional_gpus, 2)\n",
1126+
" })\n",
1127+
"\n",
1128+
"aws_instances_embodied = pd.DataFrame(aws_instances_embodied)\n",
1129+
"\n",
1130+
"# Pick some random instances to test the results are as expected\n",
1131+
"result = aws_instances_embodied.query('type == \\\"a1.medium\\\"')\n",
1132+
"assert np.isclose(result['additional_memory'], 22.21)\n",
1133+
"assert np.isclose(result['additional_storage'], 0)\n",
1134+
"assert np.isclose(result['additional_cpus'], 0)\n",
1135+
"assert np.isclose(result['additional_gpus'], 0)\n",
1136+
"assert np.isclose(result['total'], 1022.21)\n",
1137+
"\n",
1138+
"result = aws_instances_embodied.query('type == \\\"c3.xlarge\\\"')\n",
1139+
"assert np.isclose(result['additional_memory'], 61.07)\n",
1140+
"assert np.isclose(result['additional_storage'], 200.0)\n",
1141+
"assert np.isclose(result['additional_cpus'], 100.0)\n",
1142+
"assert np.isclose(result['additional_gpus'], 0)\n",
1143+
"assert np.isclose(result['total'], 1361.07)\n",
1144+
"\n",
1145+
"result = aws_instances_embodied.query('type == \\\"g4dn.xlarge\\\"')\n",
1146+
"assert np.isclose(result['additional_memory'], 510.79)\n",
1147+
"assert np.isclose(result['additional_storage'], 200.0)\n",
1148+
"assert np.isclose(result['additional_cpus'], 100.0)\n",
1149+
"assert np.isclose(result['additional_gpus'], 1200.0)\n",
1150+
"assert np.isclose(result['total'], 3010.79)"
1151+
]
1152+
},
9571153
{
9581154
"cell_type": "markdown",
9591155
"metadata": {},
9601156
"source": [
9611157
"## GCP\n",
9621158
"\n",
1159+
"### Use stage coefficients\n",
1160+
"\n",
9631161
"These values go in [`packages/gcp/src/domain/GcpFootprintEstimationConstants.ts`](https://github.com/cloud-carbon-footprint/cloud-carbon-footprint/blob/trunk/packages/gcp/src/domain/GcpFootprintEstimationConstants.ts)"
9641162
]
9651163
},

Diff for: ‎data/README.md

+2
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,8 @@ supports.
1111
- [AWS Instance Types](https://aws.amazon.com/ec2/instance-types/) based on
1212
[Teads Carbon Footprint
1313
Dataset](https://docs.google.com/spreadsheets/d/1YhtGO_UU9Hc162m7eQKYFQOnV4_yEK5_lgHYfl02JPE/edit#gid=504755275).
14+
- AWS Instance CPUs based on [Teads AWS Platforms Ratios
15+
sheet](https://docs.google.com/spreadsheets/d/1YhtGO_UU9Hc162m7eQKYFQOnV4_yEK5_lgHYfl02JPE/edit#gid=1695769209).
1416
- [Azure Machine
1517
Types](https://azure.microsoft.com/en-us/pricing/details/virtual-machines/linux/)
1618
- [GCP Machine Types](https://cloud.google.com/compute/docs/machine-types)

Diff for: ‎data/aws-instance-cpus.csv

+24
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
CPU Name,Platform Total Number of vCPU,Platform Number of CPU Socket(s),Total Number of vCPU per socket
2+
Xeon E5-2666 v3,40,2,20
3+
Xeon E5-2676 v3,48,2,24
4+
Xeon E5-2686 v4,72,2,36
5+
Xeon E5-2650,32,2,16
6+
Xeon E5-2665,32,2,16
7+
Xeon E5-2670,32,2,16
8+
Xeon E5-2651 v2,48,2,24
9+
Xeon E5-2670 v2,40,2,20
10+
Xeon E5-2680 v2,40,2,20
11+
Xeon E7-8880 v3,128,4,32
12+
Xeon Platinum 8124M,72,2,36
13+
Xeon Platinum 8151,48,2,24
14+
Xeon Platinum 8175M,96,2,48
15+
Xeon Platinum 8176M,448,8,56
16+
Xeon Platinum 8252C,48,2,24
17+
Xeon Platinum 8259CL,96,2,48
18+
Xeon Platinum 8275CL,96,2,48
19+
Xeon Platinum 8375C,128,2,64
20+
EPYC 7571,96,2,48
21+
EPYC 7R32,96,1,96
22+
Graviton,16,1,16
23+
Graviton2,64,1,64
24+
Core i7-8700B,12,1,12

0 commit comments

Comments
 (0)