Skip to content

Commit

Permalink
Read config from bootstrap.json
Browse files Browse the repository at this point in the history
  • Loading branch information
hejianfei committed Dec 1, 2024
1 parent 4b1e80a commit 01764c7
Show file tree
Hide file tree
Showing 18 changed files with 512 additions and 167 deletions.
46 changes: 29 additions & 17 deletions dubbo-demo/dubbo-demo-xds/debug-document.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,9 @@

进入到下载包所在路径,执行命令`istioctl install`进行安装。

![](https://cdn.nlark.com/yuque/0/2024/png/12647363/1732375873262-71c4641d-ba91-457a-9b3f-747743a66e90.png)
![img.png](images/1.png)



> **注意:若 Mac电脑 安装过程中提示无法校验安全性,此时先不要关闭弹出窗口,只需要打开 「设置」-「隐私与安全性」-「仍要运行」,随后再执行一次`istioctl install` 命令,就会看到一个弹窗,点击打开,即可安装。
>
> 注意:若 Mac电脑 安装过程中提示无法校验安全性,此时先不要关闭弹出窗口,只需要打开 「设置」-「隐私与安全性」-「仍要运行」,随后再执行一次`istioctl install` 命令,就会看到一个弹窗,点击打开,即可安装。
# 02 远程K8s调试示例
## 2.1 开启镜像仓库
Expand Down Expand Up @@ -54,24 +51,23 @@ mvn clean install -DskipTests

`start.sh`脚本主要完成的任务如下:

1、新建名为`dubbo-demo``namespace`,并切换到此`namespace`**
1、新建名为`dubbo-demo``namespace`,并切换到此`namespace`

**2、构建`dubbo-demo-xds-provider``dubbo-demo-xds-consumer`镜像,并推送至刚刚开启的本地镜像仓库。构建镜像时将`resource/bootstrap.json`文件拷贝至镜像 `/bootstrap.json`目录下,同时开启远程`debug`端口。**
2、构建`dubbo-demo-xds-provider``dubbo-demo-xds-consumer`镜像,并推送至刚刚开启的本地镜像仓库。构建镜像时将`resource/bootstrap.json`文件拷贝至镜像 `/bootstrap.json`目录下,同时开启远程`debug`端口。

**3、通过`service.yaml`文件,创建`k8s`资源。**
3、通过`service.yaml`文件,创建`k8s`资源。

**4、端口转发,将`istiod``15010`端口进行转发,方便本地直连`istiod`。将d`ubbo-demo-xds-consumer`服务的`31000`端口进行转发,方便远程`debug`**
4、端口转发,将`istiod``15010`端口进行转发,方便本地直连`istiod`。将d`ubbo-demo-xds-consumer`服务的`31000`端口进行转发,方便远程`debug`

## **2.4 IDEA开启远程debug**
运行`start.sh`脚本后,通过`Docker Desktop`查看对应`pod`日志,可以看到`dubbo-demo-xds-provider`服务会自动运行,而`dubbo-demo-xds-**consumer`服务暂时挂起,等待调试中。此时需要编辑本地`idea调试配置`,增加断点,即可开始调试。
## 2.4 IDEA开启远程debug
运行`start.sh`脚本后,通过`Docker Desktop`查看对应`pod`日志,可以看到`dubbo-demo-xds-provider`服务会自动运行,而`dubbo-demo-xds-consumer`服务暂时挂起,等待调试中。此时需要编辑本地`idea调试配置`,增加断点,即可开始调试。

1、编辑调试配置
**1、编辑调试配置**

![](https://cdn.nlark.com/yuque/0/2024/png/12647363/1732984798806-18f77ef3-7f8f-4a10-a03c-721b5f8002fc.png)
![img.png](images/2.png)
**2、新增`Remote JVM Debug`类型的配置,端口设置为`31000``module`选择`dubbo-demo-xds-consumer`**

2、新增`Remote JVM Debug`类型的配置,端口设置为`31000``module`选择`dubbo-demo-xds-consumer`

![](https://cdn.nlark.com/yuque/0/2024/png/12647363/1732984890317-ff1bb796-010b-4ccb-896d-e7ecfdf44343.png)
![img.png](images/3.png)

**3、新增断点后,点击调试按钮,即可进行远程调试。**

Expand All @@ -81,7 +77,23 @@ mvn clean install -DskipTests

1、`dubbo-demo-xds-consumer`服务挂起的原因是因为通过`service.yaml`文件部署资源时设置了`suspen=y`,如果仅仅是运行示例,不需要调试,可以修改为`suspend=n`,编译代码后,重新执行`start.sh`进行部署,此时会看到两个服务都会启动。

![](https://cdn.nlark.com/yuque/0/2024/png/12647363/1732985564723-ff3e9d9b-189b-45a5-9e49-6ba4c20076e5.png)
![img.png](images/4.png)

2、对于开发人员,每次修改`dubbo-xds`模块代码后,都需要重新执行`mvn spotless:apply`代码格式化,然后执行`mvn clean install -DskipTests`编译打包,最后执行`./start.sh`构建镜像,重新部署容器。

# 03 本地调试
上面的示例我们将`provider``consumer`服务都部署在了`K8s`中进行远程调试,但是这样有个缺点:一旦更改了`dubbo-xds`模块中的代码,都需要重新编译打包整个项目,耗时较长。

所以现在介绍一种效率更高的开发方法,修改代码后直接点击调试即可,不需要重新编译打包部署。

但这种方式只能用于调试资源加载过程,实际调用k8s中的provider会因为网络访问不到而失败。

原理:仍然在`k8s`中部署`provider`服务,但是`consumer`服务在本地`IDEA`中进行启动,同时转发`istiod`服务的`15010`端口,确保可以从`istiod`中获取`xds`资源。

整体步骤如下:
1、还是运行`./start.sh``provider`服务部署到`k8s`环境中,并且转发了`istiod`服务的`15010`端口。
2、修改本地`dubbo-demo-xds-consumer/src/resource/bootstrap.json` 文件,修改`server_uri为localhost:15010`
3、在 `XdsConsumerApplication` 启动类`main()`函数中设置环境变量指明`bootstrap.json`所在路径, `System.setProperty("GRPC_XDS_BOOTSTRAP", "修改为自己的路径")`
4、点击调试即可进行本地调试。

![img.png](images/5.png)
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@ public static void main(String[] args) throws InterruptedException {
// System.setProperty("NAMESPACE", "dubbo-demo");
// IstioConstant.KUBERNETES_SA_PATH = "/Users/smzdm/hjf/xds/resources/token";
// System.setProperty(IstioConstant.PILOT_CERT_PROVIDER_KEY, "istiod");

// System.setProperty("GRPC_XDS_BOOTSTRAP",
// "/Users/hejianfei/code/server/dubbo/dubbo-demo/dubbo-demo-xds/dubbo-demo-xds-consumer/src/main/resources/bootstrap.json");
ConfigurableApplicationContext context = SpringApplication.run(XdsConsumerApplication.class, args);
XdsConsumerApplication application = context.getBean(XdsConsumerApplication.class);
Thread.sleep(10000);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
}
],
"node": {
"id": "sidecar~192.168.19.141~echo-v1-5764868574-whqs9.echo-grpc~echo-grpc.svc.cluster.local",
"id": "sidecar~192.168.19.141~dubbo-v4-5764868574-whqs9.dubbo-demo~dubbo-demo.svc.cluster.local",
"metadata": {
"ANNOTATIONS": {
"inject.istio.io/templates": "grpc-agent",
Expand Down Expand Up @@ -45,7 +45,7 @@
"version": "v1"
},
"MESH_ID": "cluster.local",
"NAME": "echo-v1-5859d7bc7d-wlb2d",
"NAME": "dubbo-v4-5764868574-whqs9",
"NAMESPACE": "dubbo-demo",
"NODE_NAME": "us-west-1.192.168.19.107",
"OWNER": "kubernetes://apis/apps/v1/namespaces/echo-grpc/deployments/echo-v1",
Expand Down
Binary file added dubbo-demo/dubbo-demo-xds/images/1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added dubbo-demo/dubbo-demo-xds/images/2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added dubbo-demo/dubbo-demo-xds/images/3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added dubbo-demo/dubbo-demo-xds/images/4.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added dubbo-demo/dubbo-demo-xds/images/5.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions dubbo-xds/src/main/java/org/apache/dubbo/xds/AdsObserver.java
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,9 @@ public class AdsObserver {

private final Map<String, XdsResourceType<?>> subscribedResourceTypeUrls = new HashMap<>();

public AdsObserver(URL url, Node node) {
public AdsObserver(URL url) {
this.url = url;
this.node = node;
this.node = NodeBuilder.build();
this.xdsChannel = new XdsChannel(url);
this.applicationModel = url.getOrDefaultApplicationModel();
}
Expand Down
30 changes: 15 additions & 15 deletions dubbo-xds/src/main/java/org/apache/dubbo/xds/NodeBuilder.java
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@
*/
package org.apache.dubbo.xds;

import org.apache.dubbo.common.utils.NetUtils;
import org.apache.dubbo.xds.bootstrap.BootstrapInfo;
import org.apache.dubbo.xds.bootstrap.Bootstrapper;
import org.apache.dubbo.xds.istio.IstioEnv;

import java.util.HashMap;
Expand All @@ -28,38 +29,37 @@

public class NodeBuilder {

private static final String SVC_CLUSTER_LOCAL = ".svc.cluster.local";

public static Node build() {
// String podName = System.getenv("metadata.name");
// String podNamespace = System.getenv("metadata.namespace");

String podName = IstioEnv.getInstance().getPodName();
String podNamespace = IstioEnv.getInstance().getWorkloadNameSpace();
String svcName = IstioEnv.getInstance().getIstioMetaClusterId();
BootstrapInfo bootstrapInfo = Bootstrapper.getInstance().bootstrap();
assert bootstrapInfo.getNode().getMetadata() != null;
String podId = bootstrapInfo.getNode().getId();
String podNamespace =
(String) bootstrapInfo.getNode().getMetadata().getOrDefault("NAMESPACE", "EMPTY_NAME_SPACE");
String clusterName = (String) bootstrapInfo.getNode().getMetadata().getOrDefault("CLUSTER_ID", "Kubernetes");
String generatorName = (String) bootstrapInfo.getNode().getMetadata().getOrDefault("GENERATOR", "grpc");
String saName = IstioEnv.getInstance().getServiceAccountName();

Map<String, Value> metadataMap = new HashMap<>(2);
Map<String, Value> metadataMap = new HashMap<>();

metadataMap.put(
"ISTIO_META_NAMESPACE",
Value.newBuilder().setStringValue(podNamespace).build());
metadataMap.put(
"SERVICE_ACCOUNT", Value.newBuilder().setStringValue(saName).build());

metadataMap.put("GENERATOR", Value.newBuilder().setStringValue("grpc").build());
metadataMap.put(
"NAMESPACE", Value.newBuilder().setStringValue("dubbo-demo").build());
"GENERATOR", Value.newBuilder().setStringValue(generatorName).build());
metadataMap.put(
"NAMESPACE", Value.newBuilder().setStringValue(podNamespace).build());

Struct metadata = Struct.newBuilder().putAllFields(metadataMap).build();

// id -> sidecar~ip~{POD_NAME}~{NAMESPACE_NAME}.svc.cluster.local
// cluster -> {SVC_NAME}
return Node.newBuilder()
.setMetadata(metadata)
.setId("sidecar~" + NetUtils.getLocalHost() + "~" + "dubbo-v4-5764868574-whqs9.dubbo-demo" + "~"
+ "dubbo-demo" + SVC_CLUSTER_LOCAL)
.setCluster(svcName)
.setId(podId)
.setCluster(clusterName)
.build();
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ public class PilotExchanger {

protected PilotExchanger(URL url) {
this.pollingTimeout = url.getParameter("pollingTimeout", 10);
adsObserver = new AdsObserver(url, NodeBuilder.build());
adsObserver = new AdsObserver(url);
this.applicationModel = url.getOrDefaultApplicationModel();
}

Expand Down
7 changes: 4 additions & 3 deletions dubbo-xds/src/main/java/org/apache/dubbo/xds/XdsChannel.java
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
import org.apache.dubbo.common.URL;
import org.apache.dubbo.common.logger.ErrorTypeAwareLogger;
import org.apache.dubbo.common.logger.LoggerFactory;
import org.apache.dubbo.xds.bootstrap.BootstrapInfo;
import org.apache.dubbo.xds.bootstrap.Bootstrapper;
import org.apache.dubbo.xds.security.api.CertPair;
import org.apache.dubbo.xds.security.api.CertSource;
Expand Down Expand Up @@ -90,11 +91,11 @@ public XdsChannel(URL url) {
.build();
}
} else {
Bootstrapper bootstrapper = new Bootstrapper();
Bootstrapper.BootstrapInfo bootstrapInfo = bootstrapper.bootstrap();
BootstrapInfo bootstrapInfo = Bootstrapper.getInstance().bootstrap();
String server = bootstrapInfo.getServers().get(0).getTarget();
// URLAddress address = URLAddress.parse(bootstrapInfo.getServers().get(0).getTarget(), null, false);
// EpollEventLoopGroup elg = new EpollEventLoopGroup();
managedChannel = NettyChannelBuilder.forAddress("istiod.istio-system.svc", 15010)
managedChannel = NettyChannelBuilder.forTarget(server)
// .eventLoopGroup(elg)
// .channelType(EpollDomainSocketChannel.class)
.usePlaintext()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
*/
package org.apache.dubbo.xds;

public final class XdsInitializationException extends Exception {
public final class XdsInitializationException extends RuntimeException {

public XdsInitializationException(String message) {
super(message);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.apache.dubbo.xds.bootstrap;

import org.apache.dubbo.xds.bootstrap.Bootstrapper.AuthorityInfo;
import org.apache.dubbo.xds.bootstrap.Bootstrapper.CertificateProviderInfo;
import org.apache.dubbo.xds.bootstrap.Bootstrapper.ServerInfo;

import javax.annotation.Nullable;

import java.util.List;
import java.util.Map;

import com.google.common.collect.ImmutableList;
import com.google.common.collect.ImmutableMap;

public class BootstrapInfo {
private final ImmutableList<ServerInfo> servers;
private final Node node;

@Nullable
private final ImmutableMap<String, CertificateProviderInfo> certProviders;

@Nullable
private final String serverListenerResourceNameTemplate;

private final String clientDefaultListenerResourceNameTemplate;
private final ImmutableMap<String, AuthorityInfo> authorities;

private BootstrapInfo(Builder builder) {
this.servers = ImmutableList.copyOf(builder.servers);
this.node = builder.node;
this.certProviders = builder.certProviders == null ? null : ImmutableMap.copyOf(builder.certProviders);
this.serverListenerResourceNameTemplate = builder.serverListenerResourceNameTemplate;
this.clientDefaultListenerResourceNameTemplate = builder.clientDefaultListenerResourceNameTemplate;
this.authorities = builder.authorities == null ? null : ImmutableMap.copyOf(builder.authorities);
}

public ImmutableList<ServerInfo> getServers() {
return servers;
}

public Node getNode() {
return node;
}

@Nullable
public ImmutableMap<String, CertificateProviderInfo> getCertProviders() {
return certProviders;
}

@Nullable
public String getServerListenerResourceNameTemplate() {
return serverListenerResourceNameTemplate;
}

public String getClientDefaultListenerResourceNameTemplate() {
return clientDefaultListenerResourceNameTemplate;
}

public ImmutableMap<String, AuthorityInfo> getAuthorities() {
return authorities;
}

public static Builder builder() {
return new Builder();
}

public static final class Builder {
private List<ServerInfo> servers;
private Node node;
private Map<String, CertificateProviderInfo> certProviders;
private String serverListenerResourceNameTemplate;
private String clientDefaultListenerResourceNameTemplate;
private Map<String, AuthorityInfo> authorities;

public Builder servers(List<ServerInfo> servers) {
this.servers = servers;
return this;
}

public Builder node(Node node) {
this.node = node;
return this;
}

public Builder certProviders(@Nullable Map<String, CertificateProviderInfo> certProviders) {
this.certProviders = certProviders;
return this;
}

public Builder serverListenerResourceNameTemplate(@Nullable String serverListenerResourceNameTemplate) {
this.serverListenerResourceNameTemplate = serverListenerResourceNameTemplate;
return this;
}

public Builder clientDefaultListenerResourceNameTemplate(String clientDefaultListenerResourceNameTemplate) {
this.clientDefaultListenerResourceNameTemplate = clientDefaultListenerResourceNameTemplate;
return this;
}

public Builder authorities(Map<String, AuthorityInfo> authorities) {
this.authorities = authorities;
return this;
}

public BootstrapInfo build() {
return new BootstrapInfo(this);
}
}

@Override
public String toString() {
return "BootstrapInfo{" + "servers=" + servers + ", node=" + node + ", certProviders=" + certProviders
+ ", serverListenerResourceNameTemplate='" + serverListenerResourceNameTemplate + '\''
+ ", clientDefaultListenerResourceNameTemplate='" + clientDefaultListenerResourceNameTemplate + '\''
+ ", authorities=" + authorities + '}';
}
}
Loading

0 comments on commit 01764c7

Please sign in to comment.