Contents
nginx ingress 安装
参考： https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/index.md
参考： https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx
参考： https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip
参考： https://metallb.universe.tf/usage/#traffic-policies
参考： K8s Ingress、Ingress Controller 和 Ingress Class 以及转载地址 https://www.jianshu.com/p/78e27347076c
参考： https://kubernetes.github.io/ingress-nginx/user-guide/tls/
场景
基于LoadBalancer，可以安装ingress。可以提供 layer7的负载均衡等功能。
多个应用或服务统一走一个ingress，如果是云端环境，还可以减少不少费用。
参考： https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx
参考： https://kubernetes.github.io/ingress-nginx/deploy/
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update使用helm安装ingress-nginx
创建命名空间
kubectl create ns ingress-nginx切换ingress-nginx作为默认命名空间
kubectl config set-context --current --namespace ingress-nginx安装nginx ingress 套件
helm install ingress-nginx ingress-nginx/ingress-nginx显示结果
root@wangjm-B550M-K-1:~/k8s/helm/ingress-nginx# helm install ingress-nginx ingress-nginx/ingress-nginx
NAME: ingress-nginx
LAST DEPLOYED: Thu May 9 00:02:05 2024
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the load balancer IP to be available.
You can watch the status by running 'kubectl get service --namespace ingress-nginx ingress-nginx-controller --output wide --watch'
An example Ingress that makes use of the controller:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example
namespace: foo
spec:
ingressClassName: nginx
rules:
- host: www.example.com
http:
paths:
- pathType: Prefix
backend:
service:
name: exampleService
port:
number: 80
path: /
# This section is only required if TLS is to be enabled for the Ingress
tls:
- hosts:
- www.example.com
secretName: example-tls
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
apiVersion: v1
kind: Secret
metadata:
name: example-tls
namespace: foo
data:
tls.crt: <base64 encoded cert>
tls.key: <base64 encoded key>
type: kubernetes.io/tls
按上面的提示，确认nginx ingress controller已经安装好
root@wangjm-B550M-K-1:~/k8s/helm/ingress-nginx# kubectl --namespace ingress-nginx get services -o wide -w ingress-nginx-controller
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
ingress-nginx-controller LoadBalancer 172.31.7.165 192.168.1.100 80:30794/TCP,443:31041/TCP 4m55s app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
这里的external-ip是metallb分配的一个ip。 192.168.1.100
而且上面的提示给了一个ingress的例子。
参考： https://kubernetes.io/zh-cn/docs/concepts/services-networking/ingress-controllers/#using-multiple-ingress-controllers
看了下，默认的配置，并没有设置ingress-nginx作为默认的ingressClass
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
annotations:
meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ingress-nginx
creationTimestamp: "2024-05-08T16:02:15Z"
generation: 1
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.10.1
helm.sh/chart: ingress-nginx-4.10.1
name: nginx
resourceVersion: "173311"
uid: 85c3bf83-8d33-495e-8e4c-ed878a5d7956
spec:
controller: k8s.io/ingress-nginx
为了将来不必要的麻烦，这里将其设为默认的ingressClass
kubectl edit ingressclasses.networking.k8s.io nginx在metadata.annotations中添加一行ingressclass.kubernetes.io/is-default-class: "true"
如下所示
root@wangjm-B550M-K-1:~/k8s/helm/ingress-nginx# kubectl get ingressclasses.networking.k8s.io nginx -o yaml
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
annotations:
ingressclass.kubernetes.io/is-default-class: "true"
meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ingress-nginx
creationTimestamp: "2024-05-08T16:02:15Z"
generation: 1
labels:
app.kubernetes.io/component: controller
app.kubernetes.io/instance: ingress-nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/version: 1.10.1
helm.sh/chart: ingress-nginx-4.10.1
name: nginx
resourceVersion: "174859"
uid: 85c3bf83-8d33-495e-8e4c-ed878a5d7956
spec:
controller: k8s.io/ingress-nginx
可以按照前面helm安装成功后的提示， 以及ingress-nginx官方文档的demo， （https://kubernetes.github.io/ingress-nginx/deploy/），
部署一个ingress试试效果
An example Ingress that makes use of the controller:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example
namespace: foo
spec:
ingressClassName: nginx
rules:
- host: www.example.com
http:
paths:
- pathType: Prefix
backend:
service:
name: exampleService
port:
number: 80
path: /
# This section is only required if TLS is to be enabled for the Ingress
tls:
- hosts:
- www.example.com
secretName: example-tls
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
apiVersion: v1
kind: Secret
metadata:
name: example-tls
namespace: foo
data:
tls.crt: <base64 encoded cert>
tls.key: <base64 encoded key>
type: kubernetes.io/tls
关于如何生成tls用到的crt和key， 可以参考这里： https://kubernetes.github.io/ingress-nginx/user-guide/tls/
本地测试
测试例子
切换到default命名空间，作为默认操作空间
kubectl config set-context --current --namespace default参考： https://kubernetes.github.io/ingress-nginx/deploy/#local-testing，
kubectl create deployment demo --image=httpd --port=80
kubectl expose deployment demo
kubectl create ingress demo-localhost --class=nginx \
--rule="demo.localdev.me/*=demo:80"查看
[root@jingmin-kube-archlinux ingress-nginx]# kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
demo-localhost nginx demo.localdev.me 192.168.1.100 80 5s
[root@jingmin-kube-archlinux ingress-nginx]# kubectl get ingress -o yaml
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
creationTimestamp: "2023-08-17T03:10:20Z"
generation: 1
name: demo-localhost
namespace: default
resourceVersion: "146615"
uid: f13f11e3-d8e6-45f8-a15b-5f442db09a59
spec:
ingressClassName: nginx
rules:
- host: demo.localdev.me
http:
paths:
- backend:
service:
name: demo
port:
number: 80
path: /
pathType: Prefix
status:
loadBalancer:
ingress:
- ip: 192.168.1.100
kind: List
metadata:
resourceVersion: ""
注意到这里，自动带上了ingressClassName: nginx。
一方面：回想一下前面，我们修改过相关配置kubectl edit ingressclasses.networking.k8s.io nginx，将这个IngressClass作为了默认的ingressClassName。
另一方面： 这里创建ingress 时，指定了选项--class=nginx 。
请求测试结果：
[root@jingmin-kube-archlinux ingress-nginx]# curl --resolve demo.localdev.me:80:192.168.1.100 http://demo.localdev.me
<html><body><h1>It works!</h1></body></html>
生成tls证书
参考： https://kubernetes.github.io/ingress-nginx/user-guide/tls/
参考： https://kubernetes.github.io/ingress-nginx/examples/auth/client-certs/
参考： https://kubernetes.io/zh-cn/docs/tasks/administer-cluster/certificates/
参考： https://serverfault.com/questions/466683/can-an-ssl-certificate-be-on-a-single-line-in-a-file-no-line-breaks
参考： https://zhuanlan.zhihu.com/p/572539788
cd /tmp
mkdir -p /tmp/demo-localdev-me-tls
cd /tmp/demo-localdev-me-tls
[root@jingmin-kube-archlinux tmp_20230817]# openssl genrsa -out server.key 1024
[root@jingmin-kube-archlinux tmp_20230817]# ls
server.key
[root@jingmin-kube-archlinux tmp_20230817]# cat server.key
...
[root@jingmin-kube-archlinux tmp_20230817]# openssl req -new -key server.key -out server.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CN
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:demo.localdev.me
Email Address []:784319947@163.com
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[root@jingmin-kube-archlinux demo-localdev-me-tls]# ls
server.csr server.key
[root@jingmin-kube-archlinux demo-localdev-me-tls]# openssl x509 -req -in server.csr -out server.crt -signkey server.key -days 3650
...
[root@jingmin-kube-archlinux demo-localdev-me-tls]# ls
server.crt server.csr server.key
[root@jingmin-kube-archlinux demo-localdev-me-tls]# cat server.crt
[root@jingmin-kube-archlinux demo-localdev-me-tls]# cat server.key
[root@jingmin-kube-archlinux demo-localdev-me-tls]# ls
server.crt server.csr server.key
[root@jingmin-kube-archlinux demo-localdev-me-tls]# kubectl create secret tls demo-localdev-me-tls --key ./server.key --cert ./server.crt
[root@jingmin-kube-archlinux demo-localdev-me-tls]# kubectl get secrets demo-localdev-me-tls -o yaml
apiVersion: v1
data:
tls.crt: ...
tls.key: ...
kind: Secret
metadata:
creationTimestamp: "2023-08-17T04:13:01Z"
name: demo-localdev-me
namespace: default
resourceVersion: "154361"
uid: 7cbc8ab1-6e3f-4ec3-b384-51bfc78c04d2
type: kubernetes.io/tls
然后编辑ingress， 添加tls部分
[root@jingmin-kube-archlinux demo-localdev-me-tls]# kubectl edit ingress demo-localhost
[root@jingmin-kube-archlinux demo-localdev-me-tls]# kubectl get ingress demo-localhost -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
creationTimestamp: "2023-08-17T03:10:20Z"
generation: 2
name: demo-localhost
namespace: default
resourceVersion: "155578"
uid: f13f11e3-d8e6-45f8-a15b-5f442db09a59
spec:
ingressClassName: nginx
rules:
- host: demo.localdev.me
http:
paths:
- backend:
service:
name: demo
port:
number: 80
path: /
pathType: Prefix
tls:
- hosts:
- demo.localdev.me
secretName: demo-localdev-me-tls
status:
loadBalancer:
ingress:
- ip: 192.168.1.100
使用https请求测试(这里需要注意端口是443，且curl加选项允许不安全的访问)
[root@jingmin-kube-archlinux tmp_20230817]# curl --resolve demo.localdev.me:443:192.168.1.100 https://demo.localdev.me --insecure
<html><body><h1>It works!</h1></body></html>
[root@jingmin-kube-archlinux tmp_20230817]# for i in {1..10} ; do curl --resolve demo.localdev.me:443:192.168.1.100 https://demo.localdev.me --insecure; sleep 1; done
<html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>
<html><body><h1>It works!</h1></body></html>
外部访问测试
外部访问测试(暂无解释 todo)
kubectl create ingress demo-external --class=nginx --rule="demo.ole12138.cn/*=demo:80"大概是这样的访问流程：公网访问阿里云备案的域名和服务器，经过了几次转发，到达内网的k8s
注意，这次是为泛域名*.ole12138.cn生成tls证书
mkdir /tmp/any-ole12138-cn-tls
cd /tmp/any-ole12138-cn-tls
[root@jingmin-kube-archlinux tmp_20230817_2]# openssl genrsa -out server.key 1024
[root@jingmin-kube-archlinux tmp_20230817_2]# openssl req -new -key server.key -out server.csr
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CN
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:*.ole12138.cn
Email Address []:784319947@qq.com
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
[root@jingmin-kube-archlinux tmp_20230817_2]# ls
server.csr server.key
[root@jingmin-kube-archlinux tmp_20230817_2]# openssl x509 -req -in server.csr -out server.crt -signkey server.key -days 3650
[root@jingmin-kube-archlinux tmp_20230817_2]# kubectl create secret tls any-ole12138-cn-tls --key ./server.key --cert ./server.crt
[root@jingmin-kube-archlinux tmp_20230817_2]# kubectl edit ingress demo-external
#修改tls部分
[root@jingmin-kube-archlinux tmp_20230817_2]# kubectl get ingress demo-external -o yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
creationTimestamp: "2023-08-17T05:04:49Z"
generation: 2
name: demo-external
namespace: default
resourceVersion: "164607"
uid: 416dafa7-1825-4602-abd6-18de1b4066ae
spec:
ingressClassName: nginx
rules:
- host: demo.ole12138.cn
http:
paths:
- backend:
service:
name: demo
port:
number: 80
path: /
pathType: Prefix
tls:
- hosts:
- demo.ole12138.cn
secretName: any-ole12138-cn-tls
status:
loadBalancer:
ingress:
- ip: 192.168.1.100
for i in {1..10} ; do curl --resolve demo.ole12138.cn:443:192.168.1.100 https://demo.ole12138.cn --insecure; sleep 1; done查看日志
参考： https://github.com/kubernetes/ingress-nginx/pull/1840
参考： https://github.com/kubernetes/ingress-nginx/pull/1840/commits/578e7a0adb5d95e4b6bfbd790296f3d4b6ca1088
Access log path. Goes to
/var/log/nginx/access.logby default. 访问日志路径。默认情况下转到“/var/log/nginx/access.log”。
Note: the file
/var/log/nginx/access.logis a symlink to/dev/stdout注意： 文件/var/log/nginx/access.log是/dev/stdout的符号链接
Error log path. Goes to
/var/log/nginx/error.logby default. 错误日志路径。默认情况下转到“/var/log/nginx/error.log”。
Note: the file
/var/log/nginx/error.logis a symlink to/dev/stderr注意： 文件/var/log/nginx/error.log是/dev/stderr的符号链接
所以默认情况下，直接 kubectl logs命令查看controller日志即可。
[root@jingmin-kube-archlinux ~]# kubectl -n ingress-nginx logs pods/ingress-nginx-controller-5fcb5746fc-xlj5s
[root@jingmin-kube-archlinux ~]# kubectl -n ingress-nginx logs pods/ingress-nginx-controller-5fcb5746fc-xlj5s -f可以发现，这种方式，对于业务场景中排查问题，还是不太方便的。
如果是云端厂商提供的服务，可能会配套提供ingress相关的日志插件。我们私网场景下可以考虑使用elk之类的日志栈。
添加自定义配置, 重新部署
helm upgrade ingress-nginx -f my-override-values.yaml ingress-nginx/ingress-nginx输出
root@wangjm-B550M-K-1:~/k8s/helm/ingress-nginx# vim my-override-values.yaml
root@wangjm-B550M-K-1:~/k8s/helm/ingress-nginx# helm upgrade ingress-nginx -f my-override-values.yaml ingress-nginx/ingress-nginx
Release "ingress-nginx" has been upgraded. Happy Helming!
NAME: ingress-nginx
LAST DEPLOYED: Fri May 31 14:55:48 2024
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 3
TEST SUITE: None
NOTES:
The ingress-nginx controller has been installed.
It may take a few minutes for the load balancer IP to be available.
You can watch the status by running 'kubectl get service --namespace ingress-nginx ingress-nginx-controller --output wide --watch'
An example Ingress that makes use of the controller:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example
namespace: foo
spec:
ingressClassName: nginx
rules:
- host: www.example.com
http:
paths:
- pathType: Prefix
backend:
service:
name: exampleService
port:
number: 80
path: /
# This section is only required if TLS is to be enabled for the Ingress
tls:
- hosts:
- www.example.com
secretName: example-tls
If TLS is enabled for the Ingress, a Secret containing the certificate and key must also be provided:
apiVersion: v1
kind: Secret
metadata:
name: example-tls
namespace: foo
data:
tls.crt: <base64 encoded cert>
tls.key: <base64 encoded key>
type: kubernetes.io/tls
发表回复