tonglin0325的个人主页

Zk学习笔记——创建节点

tonglin0325

参考：从Paxos到Zookeeper分布式一致性原理和实践

使用的zk依赖是cdh5.16.2的3.4.5

1
2
3
4
5
6
7
<!-- zookeeper -->
<dependency>
    <groupId>org.apache.zookeeper</groupId>
    <artifactId>zookeeper</artifactId>
    <version>3.4.5-cdh5.16.2</version>
</dependency>

Zk在创建节点的有同步创建和异步创建，但是Zk是不支持递归创建节点，即父节点必须存储。

在创建同名的节点的时候，会抛出NodeExistsException异常。

Zookeeper节点内容只支持字节数组（byte[]）

同步创建节点代码，其中创建了一个永久节点，key是/app1，value是123

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
package com.bigdata.zookeeper;


import org.apache.zookeeper.*;
import java.util.concurrent.CountDownLatch;

public class ZkExample implements Watcher {

    public static CountDownLatch connectedSemaphore = new CountDownLatch(1);

    public static void main(String[] args) throws Exception {
        ZooKeeper zk = new ZooKeeper("master:2181", 5000, new ZkExample());
        System.out.println(zk.getState());
        try {
            connectedSemaphore.await();
            String path = zk.create("/app1", "123".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
            System.out.println("success create znode: " + path);
        } catch (InterruptedException e) {
            System.out.println("Zk session established" + e);
        }
    }

    @Override
    public void process(WatchedEvent watchedEvent) {
        System.out.println(watchedEvent);
        if (Event.KeeperState.SyncConnected == watchedEvent.getState()) {
            connectedSemaphore.countDown();
        }
    }
}

输出，使用Zkui查看zk中的数据，能发现多出了/app1的节点，节点的值是123

全文 >>

Zk学习笔记——读取节点

tonglin0325

参考：从Paxos到Zookeeper分布式一致性原理和实践

使用的zk依赖是cdh5.16.2的3.4.5

1
2
3
4
5
6
7
<!-- zookeeper -->
<dependency>
    <groupId>org.apache.zookeeper</groupId>
    <artifactId>zookeeper</artifactId>
    <version>3.4.5-cdh5.16.2</version>
</dependency>

代码，在初始化zk的时候，会触发一个watchEvent，将CountDownLatch-1=0，从而开始读取/app1的value，此时为12345，

然后将其修改成123，会再次触发一个watchEvent，此时watchEvent的type是NodeDataChanged，之后10秒延时过后，输出/app1的value，此时为123

读取节点的getData放中的true表示是否注册一个watch，注意这里的watch是一次性的，触发通过之后就失效，需要反复注册watch

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
package com.bigdata.zookeeper;


import org.apache.zookeeper.AsyncCallback;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.data.Stat;

import java.util.concurrent.CountDownLatch;

public class ZkExample implements Watcher {

    public static CountDownLatch connectedSemaphore = new CountDownLatch(1);
    private static Stat stat = new Stat();

    public static void main(String[] args) throws Exception {
        ZooKeeper zk = new ZooKeeper("master:2181", 5000, new ZkExample());
        System.out.println(zk.getState());
        try {
            connectedSemaphore.await();
            System.out.println(new String(zk.getData("/app1", true, stat)));
            Thread.sleep(10000); // 10秒延时
            System.out.println(new String(zk.getData("/app1", true, stat)));
        } catch (InterruptedException e) {
            System.out.println("Zk session established" + e);
        }
    }

    @Override
    public void process(WatchedEvent watchedEvent) {
        System.out.println(watchedEvent);
        if (Event.KeeperState.SyncConnected == watchedEvent.getState()) {
            connectedSemaphore.countDown();
        }
    }
}

输出

异步代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
package com.bigdata.zookeeper;


import org.apache.zookeeper.*;
import org.apache.zookeeper.data.Stat;

import java.util.List;
import java.util.concurrent.CountDownLatch;

public class ZkExample implements Watcher {

    public static CountDownLatch connectedSemaphore = new CountDownLatch(1);
    private static Stat stat = new Stat();
    private static ZooKeeper zk = null;

    public static void main(String[] args) throws Exception {
        zk = new ZooKeeper("master:2181", 5000, new ZkExample());
        System.out.println(zk.getState());
        try {
            connectedSemaphore.await();
            // 注册一个watch，将能异步获取到为空的子节点
            zk.getChildren("/app1", true, new IChildren2Callback(), null);
            // 创建一个子节点
            zk.create("/app1/app12", "".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);

            Thread.sleep(10000); // 10秒延时

        } catch (InterruptedException e) {
            System.out.println("Zk session established" + e);
        }
    }

    @Override
    public void process(WatchedEvent watchedEvent) {
        System.out.println(watchedEvent);
        if (Event.KeeperState.SyncConnected == watchedEvent.getState()) {
            if (Event.EventType.None == watchedEvent.getType() &amp;&amp; null == watchedEvent.getPath()) {
                connectedSemaphore.countDown();
            } else if (watchedEvent.getType() == Event.EventType.NodeChildrenChanged) {
                try {
                    System.out.println(zk.getChildren(watchedEvent.getPath(), true));
                } catch (Exception e) {

                }
            }

        }
    }
}

class IChildren2Callback implements AsyncCallback.Children2Callback {

    @Override
    public void processResult(int i, String s, Object o, List<String> list, Stat stat) {
        System.out.println("Get children znode : [response code: " + i + ", path: " + s +
                ", children list" + list + ", stat : " + stat);
    }
}

输出

全文 >>

Zk学习笔记——连接Zookeeper

tonglin0325

参考：从Paxos到Zookeeper分布式一致性原理和实践

使用的zk依赖是cdh5.16.2的3.4.5

1
2
3
4
5
6
7
<!-- zookeeper -->
<dependency>
    <groupId>org.apache.zookeeper</groupId>
    <artifactId>zookeeper</artifactId>
    <version>3.4.5-cdh5.16.2</version>
</dependency>

代码，其中CountDownLatch计数器参考：Java多线程——其他工具类CyclicBarrier、CountDownLatch和Exchange

初始化的计数器为1，当zk client连接到zk集群后，zk集群返回的状态为connected，计数器-1，触发InterruptedException

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
package com.bigdata.zookeeper;

import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;

import java.util.concurrent.CountDownLatch;

public class ZkExample implements Watcher {

    public static CountDownLatch connectedSemaphore = new CountDownLatch(1);

    public static void main(String[] args) throws Exception {
        ZooKeeper zk = new ZooKeeper("master:2181", 5000, new ZkExample());
        System.out.println(zk.getState());
        try {
            connectedSemaphore.await();
        } catch (InterruptedException e) {
            System.out.println("Zk session established");
        }
    }

    @Override
    public void process(WatchedEvent watchedEvent) {
        System.out.println(watchedEvent);
        if (Event.KeeperState.SyncConnected == watchedEvent.getState()) {
            connectedSemaphore.countDown();
        }
    }
}

输出

全文 >>

小米手机卸载小爱同学

tonglin0325

1.将系统进行root

手机的miui版本需要是开发版，并且进行了root，这样才能获取到卸载系统自带软件的权限

尤其是小爱同学这个语音助手，消耗了过多的系统资源，使得手机变得卡顿，所以建议卸载掉

像通过安全中心进行卸载的方式是不能完全卸载掉的，会随着系统后台再次进行安装，所以只能获取root权限后，使用root权限进行删除

如果需要获取root权限，就需要刷开发版的rom，下载ROM请去

1
2
https://xiaomirom.com/

2.在手机上安装termux app

这个app用于在手机上执行终端命令

1
2
https://github.com/termux/termux-app/releases

3.手机安装termux-adb

这个用于在termux上连上手机adb模式，开启调试

参考：https://github.com/MasterDevX/Termux-ADB

1
2
apt update &amp;&amp; apt install wget &amp;&amp; wget https://github.com/MasterDevX/Termux-ADB/raw/master/RemoveTools.sh &amp;&amp; bash RemoveTools.sh

4.使用pm uninstall命令将小爱同学卸载掉

然后如图所示在termux终端中连上adb，输入命令进行卸载操作，就可以卸载小爱同学了

如果还需要卸载其他的系统应用的话，参考下面文章

1
2
https://powersee.github.io/2019/09/MIUI-adb/

卸载某些系统应用会导致系统异常，下面列出了我验证过的可以卸载的应用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
pm uninstall com.xiaomi.mimobile.noti
pm uninstall com.xiaomi.jr.security
pm uninstall -k --user 0 com.miui.voiceassist
pm uninstall -k --user 0 com.miui.yellowpage
pm uninstall -k --user 0 com.miui.bugreport
pm uninstall -k --user 0 com.miui.hybrid
pm uninstall -k --user 0 com.xiaomi.ab
pm uninstall -k --user 0 com.xiaomi.vipaccount
pm uninstall -k --user 0 com.miui.milivetalk
pm uninstall -k --user 0 com.xiaomi.payment
pm uninstall -k --user 0 com.miui.hybrid.accessory
pm uninstall -k --user 0 com.sohu.inputmethod.sogou.xiaomi
pm uninstall -k --user 0 com.xiaomi.gamecenter
pm uninstall -k --user 0 com.miui.analytics
pm uninstall -k --user 0 com.mipay.wallet
pm uninstall -k --user 0 com.xiaomi.joyose
pm uninstall -k --user 0 com.xiaomi.miplay
pm uninstall -k --user 0 com.miui.video
pm uninstall -k --user 0 com.miui.klo.bugreport
pm uninstall -k --user 0 com.baidu.input_mi
pm uninstall -k --user 0 com.xiaomi.pass
pm uninstall -k --user 0 com.miui.virtualsim
pm uninstall -k --user 0 com.miui.player
pm uninstall -k --user 0 com.miui.compass
pm uninstall -k --user 0 com.miui.systemAdSolution
pm uninstall -k --user 0 com.android.browser
pm uninstall -k --user 0 com.miui.contentextension
pm uninstall -k --user 0 com.xiaomi.gamecenter.sdk.service

全文 >>

使用frp+ss访问内网机器服务

tonglin0325

1.在使用frp进行内网穿透的基础上，在内网机器的frpc.ini配置中添加

1
2
3
4
5
6
[web]
type = tcp
local_ip = master
local_port = 内网端口
remote_port = 外网机器端口

启动

1
2
./frpc -c frpc.ini

2.在内网机器上启动ss-server

修改配置/etc/shadow$ocks-libev/config.json，必须是0.0.0.0

1
2
3
4
5
6
7
8
9
{
    "server":"0.0.0.0",
    "server_port":内网端口,
    "local_port":1080,
    "password":"密码",
    "timeout":60,
    "method":"chacha20-ietf-poly1305"
}

启动

1
2
ss-server -c /etc/shadow$ocks-libev/config.json

3.配置代理

全文 >>

使用frp进行内网穿透

tonglin0325

1.前提：1台有公网ip的服务器（1核1G），1台在内网的服务器（16G）

2.在公网机器上安装frp，并启动frp server

下载并解压

1
2
wget https://github.com/fatedier/frp/releases/download/v0.33.0/frp_0.33.0_linux_amd64.tar.gz

配置文件frps.ini

1
2
3
4
[common]
bind_port = xxxx
token = ssssss

其中bind_port是用于和client端通信的；token是密码；vhost_http_port是当client端配置了web http的服务的时候，通过server访问的端口；vhost_https_port是当client端配置了web https的服务的时候，通过server访问的端口

启动

1
2
./frps -c frps.ini

全文 >>

Hive学习笔记——hive hook

tonglin0325

Hive hook是hive的钩子函数，可以嵌入HQL执行的过程中运行，比如下面的这几种情况

参考

1
2
https://www.slideshare.net/julingks/apache-hive-hooksminwookim130813

有了Hook，可以实现例如非法SQL拦截，SQL收集和审计等功能，业界的案例可以参考Airbnb的reair

1
2
https://github.com/airbnb/reair

该项目中就使用了Hive的hook函数实现了一个Audit Log Hook，将提交到hiveserver2上的query写入到MySQL当中收集起来

1
2
https://github.com/airbnb/reair/blob/master/hive-hooks/src/main/java/com/airbnb/reair/hive/hooks/CliAuditLogHook.java

全文 >>

tonglin0325

标签

标签云

AWS Airbnb Android CDH Doris ELK Flink Git Grafana HAProxy HBase Hadoop Hive Hudi InfluxDB JDBC JMeter JVM JavaSE JavaWeb LaTex Linkedin Linux ML ML Infra MPP Maven MySQL Nexus OpenTSDB Paper Play Prometheus Python React Redis RocksDB Scala Solr Spark SpringBoot SpringMVC Thrift YARN antlr arthas avro cassandra clickhouse confluent docker filebeat flume golang google gradle hexo impala jenkins k8s kafka kerberos kudu ldap mac mongo mybatis nginx nlp open-falcon parquet presto ranger scyllaDB twitter zookeeper 前端 图存储及计算 多线程 广告系统 开发工具 数据结构 杂谈 正则表达式 算法 系统设计 计算机网络 设计模式

文章归档

最近文章