背景

某线上环境需将http://10.168.1.116:10020访问切换为https://10.168.1.116:10020访问，切换后某个服务启动 nacos报错:user not found情况。nacos2.0采用了grpc通信，对应grpc端口也是正常访问，该报错也是在通过grpc拉取配置时出现。

该服务关于nacos的配置如下

spring:
    application:
        name: demo-service
    config:
        import: nacos:demo-service.yml
    cloud:
        nacos:
            discovery:
                server-addr: 10.168.1.116:10020
                namespace: demo
                group: default
                password: 1234
                username: nacos
            #                enabled: false
            config:
                import-check:
                    enabled: false
                server-addr: 10.168.1.116:10020
                namespace: demo
                group: default
                password:1234
                username: nacos
                file-extension: yml
                extension-configs[0]:
                    data-id: global.yml
                    refresh: true

出现问题原因

经过本地复现及debug发现，nacos2.0客户端通过grpc访问nacos服务端前需要通过http/https的方式获取到accessToken，在本案例中未能成功拿到accessToken，grpc访问时未能携带accessToken访问nacos服务端。最终出现user not found 异常。

对我来说干扰到我定位问题因素有

• 先入为主认为nacos2.0客户端采用了grpc通信 ，应该就不会采用http/https了。所以在本次背景中的http->https切换不应该受到影响
• 网络问题为啥会报一个鉴权问题
• 最后分析出原因后，疑惑未能成功拿到accessToken http/https请求未能返回正常结果 nacos为啥不直接抛出异常

为什么没有拿到accessToken

• 我这里的配置是server-addr: 10.168.1.116:10020 。nacos会默认加上http的前缀。暴露的端口是https的就没有拿到token
• 如果配置加上https。同样也拿不到token,因为没有指定https的证书 所以还需在配置里指定证书。

我的解决方法

前提：为了减少改动所以不修改现有配置文件格式，保留nacos默认追加http前缀。

解决方法：直接修改nacos对应源码，让NacosRestTemplate 忽略证书,修改后的源码放到自己工程中 jvm会优先加载我们这个class。我这边的改动是忽略https证书并在执行请求时替换为https。

注：如果在配置文件里直接加了https前缀，直接忽略证书即可

/*
 * Copyright 1999-2018 Alibaba Group Holding Ltd.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package com.alibaba.nacos.common.http;

import com.alibaba.nacos.common.executor.NameThreadFactory;
import com.alibaba.nacos.common.http.client.NacosAsyncRestTemplate;
import com.alibaba.nacos.common.http.client.NacosRestTemplate;
import com.alibaba.nacos.common.http.client.request.DefaultAsyncHttpClientRequest;
import com.alibaba.nacos.common.http.client.request.JdkHttpClientRequest;
import com.alibaba.nacos.common.http.client.response.HttpClientResponse;
import com.alibaba.nacos.common.model.RequestHttpEntity;
import com.alibaba.nacos.common.tls.SelfHostnameVerifier;
import com.alibaba.nacos.common.tls.TlsFileWatcher;
import com.alibaba.nacos.common.tls.TlsHelper;
import com.alibaba.nacos.common.tls.TlsSystemConfig;
import org.apache.commons.lang3.StringUtils;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.config.Registry;
import org.apache.http.config.RegistryBuilder;
import org.apache.http.conn.ssl.DefaultHostnameVerifier;
import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
import org.apache.http.impl.nio.client.HttpAsyncClients;
import org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager;
import org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor;
import org.apache.http.impl.nio.reactor.IOReactorConfig;
import org.apache.http.nio.conn.NHttpClientConnectionManager;
import org.apache.http.nio.conn.NoopIOSessionStrategy;
import org.apache.http.nio.conn.SchemeIOSessionStrategy;
import org.apache.http.nio.conn.ssl.SSLIOSessionStrategy;
import org.apache.http.nio.reactor.IOReactorException;
import org.apache.http.nio.reactor.IOReactorExceptionHandler;
import org.apache.http.protocol.RequestContent;
import org.apache.http.ssl.SSLContexts;
import org.slf4j.Logger;
import org.springframework.util.Assert;

import javax.net.ssl.*;
import java.io.IOException;
import java.net.URI;
import java.security.KeyManagementException;
import java.security.NoSuchAlgorithmException;
import java.security.SecureRandom;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import java.util.Properties;
import java.util.ResourceBundle;
import java.util.function.BiConsumer;

import static com.alibaba.nacos.common.utils.ResourceUtils.getResourceAsStream;

/**
 * AbstractHttpClientFactory Let the creator only specify the http client config.
 *
 * @author mai.jh
 */
public abstract class AbstractHttpClientFactory implements HttpClientFactory {

    private static final String ASYNC_THREAD_NAME = "nacos-http-async-client";

    private static final String ASYNC_IO_REACTOR_NAME = ASYNC_THREAD_NAME + "#I/O Reactor";

    @Override
    public NacosRestTemplate createNacosRestTemplate() {
        //加入的代码 start
        //获取配置 判断当前平台是否是https访问
        //目前nacos ip_addr配的是ip,nacos会自动追加为http，在没有http https前缀的情况下

        String gateway = System.getenv("PLATFORM_GATEWAY");
        if(StringUtils.isEmpty(gateway)){
            gateway = System.getProperty("PLATFORM_GATEWAY");
        }
        Assert.notNull(gateway , "PLATFORM_GATEWAY is null");
        boolean isHttps = gateway.startsWith("https");
        HttpClientConfig httpClientConfig = buildHttpClientConfig();
        JdkHttpClientRequest clientRequest = new CustomJdkHttpClientRequest(httpClientConfig,isHttps);
        if(isHttps) {
            SSLContext context = null;
            try {
                context = SSLContext.getInstance("TLS");
            } catch (NoSuchAlgorithmException e) {
                throw new RuntimeException(e);
            }
            /* 初始化证书 */
            try {
                context.init(null, new TrustManager[] {
                        new X509TrustManager() {
                            @Override
                            public X509Certificate[] getAcceptedIssuers() {return new X509Certificate[0];}
                            @Override
                            public void checkServerTrusted(X509Certificate[] chain, String authType) throws CertificateException {}
                            @Override
                            public void checkClientTrusted(X509Certificate[] chain, String authType) throws CertificateException {}
                        }
                }, new SecureRandom());
            } catch (KeyManagementException e) {
                throw new RuntimeException(e);
            }
            /* 域名默认成功 */
            clientRequest.setSSLContext(context);
            clientRequest.replaceSSLHostnameVerifier((new HostnameVerifier() {
                @Override
                public boolean verify(String hostname, SSLSession session) {return true;}
            }));
        }
        //end
        // enable ssl
        initTls((sslContext, hostnameVerifier) -> {
            clientRequest.setSSLContext(loadSSLContext());
            clientRequest.replaceSSLHostnameVerifier(hostnameVerifier);
        }, filePath -> clientRequest.setSSLContext(loadSSLContext()));

        return new NacosRestTemplate(assignLogger(), clientRequest);
    }



    @Override
    public NacosAsyncRestTemplate createNacosAsyncRestTemplate() {
        final HttpClientConfig originalRequestConfig = buildHttpClientConfig();
        final DefaultConnectingIOReactor ioreactor = getIoReactor(ASYNC_IO_REACTOR_NAME);
        final RequestConfig defaultConfig = getRequestConfig();
        final NHttpClientConnectionManager connectionManager = getConnectionManager(originalRequestConfig, ioreactor);
        monitorAndExtension(connectionManager);
        return new NacosAsyncRestTemplate(assignLogger(), new DefaultAsyncHttpClientRequest(
                HttpAsyncClients.custom().addInterceptorLast(new RequestContent(true))
                        .setThreadFactory(new NameThreadFactory(ASYNC_THREAD_NAME))
                        .setDefaultIOReactorConfig(getIoReactorConfig()).setDefaultRequestConfig(defaultConfig)
                        .setMaxConnTotal(originalRequestConfig.getMaxConnTotal())
                        .setMaxConnPerRoute(originalRequestConfig.getMaxConnPerRoute())
                        .setUserAgent(originalRequestConfig.getUserAgent())
                        .setConnectionManager(connectionManager).build(),
                ioreactor, defaultConfig));
    }

    private DefaultConnectingIOReactor getIoReactor(String threadName) {
        final DefaultConnectingIOReactor ioreactor;
        try {
            ioreactor = new DefaultConnectingIOReactor(getIoReactorConfig(), new NameThreadFactory(threadName));
        } catch (IOReactorException e) {
            assignLogger().error("[NHttpClientConnectionManager] Create DefaultConnectingIOReactor failed", e);
            throw new IllegalStateException();
        }

        // if the handle return true, then the exception thrown by IOReactor will be ignore, and will not finish the IOReactor.
        ioreactor.setExceptionHandler(new IOReactorExceptionHandler() {

            @Override
            public boolean handle(IOException ex) {
                assignLogger().warn("[NHttpClientConnectionManager] handle IOException, ignore it.", ex);
                return true;
            }

            @Override
            public boolean handle(RuntimeException ex) {
                assignLogger().warn("[NHttpClientConnectionManager] handle RuntimeException, ignore it.", ex);
                return true;
            }
        });

        return ioreactor;
    }

    /**
     * create the {@link NHttpClientConnectionManager}, the code mainly from {@link HttpAsyncClientBuilder#build()}. we
     * add the {@link IOReactorExceptionHandler} to handle the {@link IOException} and {@link RuntimeException} thrown
     * by the {@link org.apache.http.impl.nio.reactor.BaseIOReactor} when process the event of Network. Using this way
     * to avoid the {@link DefaultConnectingIOReactor} killed by unknown error of network.
     *
     * @param originalRequestConfig request config.
     * @param ioreactor             I/O reactor.
     * @return {@link NHttpClientConnectionManager}.
     */
    private NHttpClientConnectionManager getConnectionManager(HttpClientConfig originalRequestConfig,
                                                              DefaultConnectingIOReactor ioreactor) {
        SSLContext sslcontext = SSLContexts.createDefault();
        HostnameVerifier hostnameVerifier = new DefaultHostnameVerifier();
        SchemeIOSessionStrategy sslStrategy = new SSLIOSessionStrategy(sslcontext, null, null, hostnameVerifier);

        Registry<SchemeIOSessionStrategy> registry = RegistryBuilder.<SchemeIOSessionStrategy>create()
                .register("http", NoopIOSessionStrategy.INSTANCE).register("https", sslStrategy).build();
        final PoolingNHttpClientConnectionManager poolingmgr = new PoolingNHttpClientConnectionManager(ioreactor,
                registry);

        int maxTotal = originalRequestConfig.getMaxConnTotal();
        if (maxTotal > 0) {
            poolingmgr.setMaxTotal(maxTotal);
        }

        int maxPerRoute = originalRequestConfig.getMaxConnPerRoute();
        if (maxPerRoute > 0) {
            poolingmgr.setDefaultMaxPerRoute(maxPerRoute);
        }
        return poolingmgr;
    }

    protected IOReactorConfig getIoReactorConfig() {
        HttpClientConfig httpClientConfig = buildHttpClientConfig();
        return IOReactorConfig.custom().setIoThreadCount(httpClientConfig.getIoThreadCount()).build();
    }

    protected RequestConfig getRequestConfig() {
        HttpClientConfig httpClientConfig = buildHttpClientConfig();
        return RequestConfig.custom().setConnectTimeout(httpClientConfig.getConTimeOutMillis())
                .setSocketTimeout(httpClientConfig.getReadTimeOutMillis())
                .setConnectionRequestTimeout(httpClientConfig.getConnectionRequestTimeout())
                .setContentCompressionEnabled(httpClientConfig.getContentCompressionEnabled())
                .setMaxRedirects(httpClientConfig.getMaxRedirects()).build();
    }

    protected void initTls(BiConsumer<SSLContext, HostnameVerifier> initTlsBiFunc,
                           TlsFileWatcher.FileChangeListener tlsChangeListener) {
        if (!TlsSystemConfig.tlsEnable) {
            return;
        }

        final HostnameVerifier hv = HttpsURLConnection.getDefaultHostnameVerifier();
        final SelfHostnameVerifier selfHostnameVerifier = new SelfHostnameVerifier(hv);

        initTlsBiFunc.accept(loadSSLContext(), selfHostnameVerifier);

        if (tlsChangeListener != null) {
            try {
                TlsFileWatcher.getInstance()
                        .addFileChangeListener(tlsChangeListener, TlsSystemConfig.tlsClientTrustCertPath,
                                TlsSystemConfig.tlsClientKeyPath);
            } catch (IOException e) {
                assignLogger().error("add tls file listener fail", e);
            }
        }
    }

    @SuppressWarnings("checkstyle:abbreviationaswordinname")
    protected synchronized SSLContext loadSSLContext() {
        try {
            return TlsHelper.buildSslContext(true);
        } catch (NoSuchAlgorithmException | KeyManagementException e) {
            assignLogger().error("Failed to create SSLContext", e);
        }
        return null;
    }

    /**
     * build http client config.
     *
     * @return HttpClientConfig
     */
    protected abstract HttpClientConfig buildHttpClientConfig();

    /**
     * assign Logger.
     *
     * @return Logger
     */
    protected abstract Logger assignLogger();

    /**
     * add some monitor and do some extension. default empty implementation, implemented by subclass
     */
    protected void monitorAndExtension(NHttpClientConnectionManager connectionManager) {
    }

    public static class CustomJdkHttpClientRequest extends JdkHttpClientRequest {

        /**
         * 访问平台是否是https
         */
        private boolean https;

        public CustomJdkHttpClientRequest(HttpClientConfig httpClientConfig,boolean https) {
            super(httpClientConfig);
            this.https = https;
        }

        @Override
        public HttpClientResponse execute(URI uri, String httpMethod, RequestHttpEntity requestHttpEntity) throws Exception {
            // 如果是https，则使用https 替换 掉http ,nacos配置直接用到ip端口 会默认追加http
            if (https && "http".equalsIgnoreCase(uri.getScheme())) {
                uri = new URI("https",uri.getUserInfo(),uri.getHost(),uri.getPort(),uri.getPath(),uri.getQuery(),uri.getFragment());
            }
            return super.execute(uri, httpMethod, requestHttpEntity);
        }
    }
}