Skip to content
This repository was archived by the owner on Feb 25, 2025. It is now read-only.

Add script to upload unified Android SDK CIPD archives and switch DEPS to use it.#29776

Merged
GaryQian merged 6 commits intoflutter:mainfrom
GaryQian:cipd
Nov 23, 2021
Merged

Add script to upload unified Android SDK CIPD archives and switch DEPS to use it.#29776
GaryQian merged 6 commits intoflutter:mainfrom
GaryQian:cipd

Conversation

@GaryQian
Copy link
Contributor

@GaryQian GaryQian commented Nov 17, 2021

Consolidate core Android SDK to one CIPD repo.

New script cold downloads all of the specified packages and uploads them to CIPD for all platforms. This script no longer requires your Android studio managed SDK directory to be modified.

@flutter-dashboard
Copy link

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat.

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

@skia-gold
Copy link

Gold has detected about 231 new digest(s) on patchset 2.
View them at https://flutter-engine-gold.skia.org/cl/github/29776

@skia-gold
Copy link

Gold has detected about 132 new digest(s) on patchset 4.
View them at https://flutter-engine-gold.skia.org/cl/github/29776

@GaryQian GaryQian added the Work in progress (WIP) Not ready (yet) for review! label Nov 18, 2021
@GaryQian GaryQian requested a review from blasten November 20, 2021 00:53
@GaryQian GaryQian changed the title [WIP] Add script to upload unified Android SDK CIPD archives and switch DEPS to use it. Add script to upload unified Android SDK CIPD archives and switch DEPS to use it. Nov 20, 2021
@GaryQian GaryQian removed the Work in progress (WIP) Not ready (yet) for review! label Nov 20, 2021
],
'condition': 'download_android_deps',
'dep_type': 'cipd',
},
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👏

@@ -0,0 +1,5 @@
platforms;android-31:platforms
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice. minor nit: what about moving to a folder. Maybe /tools/android_sdk then this file becomes packages.txt?

# This script requires depot_tools to be on path.

print_usage () {
echo "Usage: create_sdk_cipd_united_package.sh <PATH_TO_SDK_DIR> <VERSION_TAG>"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$1 could default to $ANDROID_SDK_ROOT

echo "Usage: create_sdk_cipd_united_package.sh <PATH_TO_SDK_DIR> <VERSION_TAG>"
echo " where:"
echo " - PATH_TO_SDK_DIR is the path to the sdk folder"
echo " - VERSION_TAG is the version of the package, e.g. 28r6 or 31v1"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
echo " - VERSION_TAG is the version of the package, e.g. 28r6 or 31v1"
echo " - VERSION_TAG is the tag of the cipd package, e.g. 28r6 or 31v1"

echo "Unable to find sdkmanager in the SDK directory. Please ensure cmdline-tools is installed."
exit 1
fi
sdkmanager_path="${find_results[$i]}"
Copy link

@blasten blasten Nov 20, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do we know that this is the desired sdkmanager? I think supporting the latest one from line 45 is probably sufficient

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want this script to not explicitly only support latest. For example, currently, latest is equivalent to 5.0. It is valid to use the 5.0 version or just simply use latest. But, even though these are equivalent, sdkmanager stores them under different subdirs (latest and 5.0 respectively).

Also, when 6.0 is released, latest would point to 6.0 under the latest subdir while people who explicitly use the 5.0 package would still have the old package under the 5.0 subdir. I want this tool to not break for people who use explicit version numbers.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it 👍

Comment on lines 64 to 66
temp_dir="$1/temp_cipd_android_sdk"
rm -rf $temp_dir
mkdir $temp_dir
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it could use mktemp -d -t android_sdk to create a temp directory with the android_sdk prefix.

# We copy only the relevant directories to a temporary dir
# for upload. sdkmanager creates extra files that we don't need.
array_length=${#split[@]}
for (( i=1; i<${array_length}; i++ )); do
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wouldn't this always be 2?. Assuming we are splitting by : and there's only one per line in android_sdk_packages.txt.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This actually supports multiple directories which can be specified with additional : delimiters. For example, platforms;android-31:platforms:licenses would copy both the platforms dir and the licenses dir that is generated by the platforms;android-31 package.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm

@@ -0,0 +1,98 @@
#!/bin/bash

# This script requires depot_tools to be on path.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we check that? sanity checks like this are time savers :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could, but since depot-tools can be installed in any directory, it would not be a robust check.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could look for which cipd and exit with code 1 if the string is empty

Copy link

@blasten blasten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@GaryQian GaryQian added the waiting for tree to go green This PR is approved and tested, but waiting for the tree to be green to land. label Nov 22, 2021
@fluttergithubbot
Copy link
Contributor

This pull request is not suitable for automatic merging in its current state.

@fluttergithubbot fluttergithubbot removed the waiting for tree to go green This PR is approved and tested, but waiting for the tree to be green to land. label Nov 23, 2021
@GaryQian GaryQian added the waiting for tree to go green This PR is approved and tested, but waiting for the tree to be green to land. label Nov 23, 2021
@GaryQian GaryQian merged commit 2cc8165 into flutter:main Nov 23, 2021
engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Nov 23, 2021
@alexmarkov
Copy link
Contributor

It looks like this commit causes failures on the head-head-head bot:

01:29 +55 ~1 -6: test/run_test.dart: run.dart script exits with code 1 when results are mixed [E]                                                                                                      
  Expected: <1>
    Actual: <0>
  [ stderr from test process ]
  
  [smoke_test_failure] [STDERR] Unhandled exception:
  [smoke_test_failure] [STDERR] FormatException: Missing extension byte (at offset 174)
  [smoke_test_failure] [STDERR] #0      _Utf8Decoder.convertChunked (dart:convert-patch/convert_patch.dart:1893:7)
  [smoke_test_failure] [STDERR] #1      _Utf8ConversionSink.addSlice (dart:convert/string_conversion.dart:314:28)
  [smoke_test_failure] [STDERR] #2      _Utf8ConversionSink.add (dart:convert/string_conversion.dart:310:5)
  [smoke_test_failure] [STDERR] #3      _ConverterStreamEventSink.add (dart:convert/chunked_conversion.dart:72:18)
  [smoke_test_failure] [STDERR] #4      _SinkTransformerStreamSubscription._handleData (dart:async/stream_transformers.dart:111:24)
  [smoke_test_failure] [STDERR] #5      _RootZone.runUnaryGuarded (dart:async/zone.dart:1618:10)
  [smoke_test_failure] [STDERR] #6      _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:341:11)
  [smoke_test_failure] [STDERR] #7      _BufferingStreamSubscription._add (dart:async/stream_impl.dart:271:7)
  [smoke_test_failure] [STDERR] #8      _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:733:19)
  [smoke_test_failure] [STDERR] #9      _StreamController._add (dart:async/stream_controller.dart:607:7)
  [smoke_test_failure] [STDERR] #10     _StreamController.add (dart:async/stream_controller.dart:554:5)
  [smoke_test_failure] [STDERR] #11     _Socket._onData (dart:io-patch/socket_patch.dart:2302:41)
  [smoke_test_failure] [STDERR] #12     _RootZone.runUnaryGuarded (dart:async/zone.dart:1618:10)
  [smoke_test_failure] [STDERR] #13     _BufferingStreamSubscription._sendData (dart:async/stream_impl.dart:341:11)
  [smoke_test_failure] [STDERR] #14     _BufferingStreamSubscription._add (dart:async/stream_impl.dart:271:7)
  [smoke_test_failure] [STDERR] #15     _SyncStreamControllerDispatch._sendData (dart:async/stream_controller.dart:733:19)
  [smoke_test_failure] [STDERR] #16     _StreamController._add (dart:async/stream_controller.dart:607:7)
  [smoke_test_failure] [STDERR] #17     _StreamController.add (dart:async/stream_controller.dart:554:5)
  [smoke_test_failure] [STDERR] #18     new _RawSocket.<anonymous closure> (dart:io-patch/socket_patch.dart:1830:33)
  [smoke_test_failure] [STDERR] #19     _NativeSocket.issueReadEvent.issue (dart:io-patch/socket_patch.dart:1314:14)
  [smoke_test_failure] [STDERR] #20     _microtaskLoop (dart:async/schedule_microtask.dart:40:21)
  [smoke_test_failure] [STDERR] #21     _startMicrotaskLoop (dart:async/schedule_microtask.dart:49:5)
  [smoke_test_failure] [STDERR] #22     _runPendingImmediateCallback (dart:isolate-patch/isolate_patch.dart:122:13)
  [smoke_test_failure] [STDERR] #23     _RawReceivePortImpl._handleMessage (dart:isolate-patch/isolate_patch.dart:193:5)

https://logs.chromium.org/logs/dart/buildbucket/cr-buildbucket/8829735632031053297/+/u/flutter_test_framework_tests/stdout

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

cla: yes needs tests platform-linux waiting for tree to go green This PR is approved and tested, but waiting for the tree to be green to land.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants