This is a Kubernetes Operator to manage Apache Kyuubi.
The kyuubi-operator is an important part of the NineInfra Data Platform, a curated combination of open-source projects including Spark, Flink, Minio, Kafka, ClickHouse, Kyuubi, and Superset, all working together seamlessly to provide users with a stable and user-friendly big data processing platform. Nineinfra is a full-stack data platform built on Kubernetes, capable of running on public cloud, private cloud, or on-premises environments.
You’ll need a Kubernetes cluster to run against. You can use KIND to get a local cluster for testing, or run against a remote cluster.
Note: Your controller will automatically use the current context in your kubeconfig file (i.e. whatever cluster kubectl cluster-info shows).
Install kyuubi operator by helm:
helm repo add nineinfra-charts https://nineinfra.github.io/nineinfra-charts/
kubectl create namespace kyuubi-operator
helm install kyuubi-operator nineinfra-charts/kyuubi-operator -n kyuubi-operator
kubectl get svc -n dwh |grep metastore
hive-metastore ClusterIP 10.100.90.209
And the service of hdfs is hdfs with the suffix .dwh.svc,the service of the metastore is the hive-metastore with the suffix .dwh.svc
2. Edit the cr yaml, there is a sample file like config/samples/kyuubi_v1alpha1_kyuubicluster.yaml
```yaml
apiVersion: kyuubi.nineinfra.tech/v1alpha1
kind: KyuubiCluster
metadata:
labels:
app.kubernetes.io/name: kyuubicluster
app.kubernetes.io/instance: kyuubicluster-sample
app.kubernetes.io/part-of: kyuubi-operator
app.kubernetes.io/managed-by: kustomize
app.kubernetes.io/created-by: kyuubi-operator
name: kyuubicluster-sample
spec:
kyuubiVersion: 1.8.1
kyuubiImage:
repository: nineinfra/kyuubi
tag: v1.8.1
kyuubiResource:
replicas: 1
kyuubiConf:
kyuubi.kubernetes.namespace: dwh
kyuubi.frontend.connection.url.use.hostname: 'false'
kyuubi.frontend.thrift.binary.bind.port: '10009'
kyuubi.frontend.thrift.http.bind.port: '10010'
kyuubi.frontend.rest.bind.port: '10099'
kyuubi.frontend.mysql.bind.port: '3309'
kyuubi.frontend.protocols: REST,THRIFT_BINARY
kyuubi.metrics.enabled: 'false'
clusterRefs:
- name: spark
type: spark
spark:
sparkMaster: k8s
sparkImage:
repository: nineinfra/spark
tag: v3.2.4
sparkNamespace: dwh
- name: hdfs
type: hdfs
hdfs:
coreSite:
fs.defaultFS: hdfs://hdfs.dwh.svc:9820
hdfsSite:
dfs.client.block.write.retries: '3'
- name: metastore
type: metastore
metastore:
hiveSite:
hive.metastore.uris: thrift://hive-metastore.dwh.svc:9083
hive.metastore.warehouse.dir: /user/hive/warehouse
kubectl apply -f kyuubi_v1alpha1_kyuubicluster.yaml -n kyuubi-operator
status: creationTime: “2023-09-28T06:11:18Z” exposedInfos:
kubectl exec -it kyuubicluster-sample-kyuubi-0 -n kyuubi-operator – bash
2. Run beeline command
```sh
kyuubi@kyuubicluster-sample-kyuubi-0:/opt/kyuubi$ cd bin
kyuubi@kyuubicluster-sample-kyuubi-0:/opt/kyuubi/bin$ ./beeline
Warn: Not find kyuubi environment file /opt/kyuubi/conf/kyuubi-env.sh, using default ones...
Beeline version 1.8.0-SNAPSHOT by Apache Kyuubi
beeline>
beeline> !connect jdbc:hive2://kyuubicluster-sample-kyuubi:10009
0: jdbc:hive2://kyuubicluster-sample-kyuubi:1> show databases;
+------------+
| namespace |
+------------+
| default |
| test |
+------------+
0: jdbc:hive2://kyuubicluster-sample-kyuubi:1> use test;
0: jdbc:hive2://kyuubicluster-sample-kyuubi:1> create table test3 (name string,id int);
0: jdbc:hive2://kyuubicluster-sample-kyuubi:1> insert into test3 values("kyuubi",1);
0: jdbc:hive2://kyuubicluster-sample-kyuubi:1> select * from test3;
+---------+-----+
| name | id |
+---------+-----+
| kyuubi | 1 |
+---------+-----+
Contributions are highly welcomed and appreciated.
This project aims to follow the Kubernetes Operator pattern.
It uses Controllers, which provide a reconcile function responsible for synchronizing resources until the desired state is reached on the cluster.
make install
make run
NOTE: You can also run this in one step by running: make install run
If you are editing the API definitions, generate the manifests such as CRs or CRDs using:
make manifests
NOTE: Run make --help for more information on all potential make targets
More information can be found via the Kubebuilder Documentation
Copyright 2023 nineinfra.
Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.