Kafka学习笔记——存储结构

2020-07-19

tonglin0325

1，由cdh安装的kafka的默认存储路径如图所示在/var/local/kafka/data，一般会进行修改

kafka配置参考：apache kafka系列之server.properties配置文件参数说明

路径下文件如下

如果是多个路径的话，使用,进行分隔，比如/data01/kafka/data,/data02/kafka/data，注意data的权限需要是kafka用户和kafka组

对应kafka manager上的topic

具体的topic的目录下的文件

lintong@master:/var/local/kafka/data/test_topic-0$ ls
00000000000000000187.index  00000000000000000187.snapshot   leader-epoch-checkpoint
00000000000000000187.log    00000000000000000187.timeindex

全文 >>

分布式协议——Paxos、Raft和ZAB

2020-07-18

tonglin0325

参考：分布式系统协议Paxos、Raft和ZAB

Paxos算法是一种提高分布式系统容错率的一致性算法

Paxos 算法的步骤是这样：

1.首先有两种角色，一个是“提议者”，一个是“接受者”。提议者可以向接受者提出提议，然后接受者表达意见。

2.因为存在多个提议者，如果同时表达意见会出现意见不一致的情况，所以首先需要尽快选出一个领导者，让意见统一。

3.然后领导者会给接受者发出提议，如果一个提议被大多数接受者接纳，这个提议就通过了。

Raft协议

Raft 协议的每个副本都会处于三种状态之一：Leader、Follower、Candidate。

Leader：所有请求的处理者，Leader 副本接受 client 的更新请求，本地处理后再同步至多个其他副本。

Follower：请求的被动更新者，从 Leader 接受更新请求，然后写入本地日志文件。

Candidate：如果 Follower 副本在一段时间内没有收到 Leader 副本的心跳，则判断 Leader 可能已经故障，此时启动选主过程，此时副本会变成 Candidate 状态，直到选主结束。

每一个副本都会维护一个 term，类似于一个逻辑时钟，每发生一个动作就会递增，通过比较每个提议的 term，副本会默认使用最新的 term，防止发生冲突。如果一个 Leader 或者 Candidate 发现自己的 term 不是最新的了，就会自动降级到 Follower，而如果一个 Follower 接收到低于自己当前 term 的提议，就会直接抛弃。

全文 >>

Zk学习笔记——应用场景

2020-07-12

tonglin0325

1.Master选举

在分布式系统中，需要选举一台机器作为master或者leader。

这时候，可以选择一个跟节点，比如/master，然后多台机器同时像这个节点创建一个子节点/master/lock，利用zookeeper的特性，最终只有一台机器能否创建成功，成功的那台机器就是Master；

其他机器注册watch到这个子节点，然后当master宕机的时候，其他机器就会重新开始选举。

2.分布式锁

分布式锁的场景通常是多个进程需要在某个节点保证同步，比如保证只有一个进程进到某个函数里面

下面例子中的查询余额和更新余额操作就需要用锁锁住，保证一个线程做完了查询和更新操作之后，才能有另外一个线程来做查询和更新操作

这个使用的锁称为排他锁（Exclusive Locks），又称为写锁或者独占锁。当一个事务1对数据对象1加了排他锁，那么在整个加锁期间，只允许事务1对对象1进行读取或者更新

一个账户有100元
进程A，查询账户余额，等于100元
进程B，查询账户余额，等于100元
进程B，取出100元，并更新，100-100=0
进程A，存入100元，并更新，100+100=200（此时发生错误）

其他的锁还有共享锁（Shared Locks），又称之为读锁。如果事务1对数据对象1加了共享锁，那个当前事务只能对数据对象1进行读操作，其他事务也只能对这个对象加共享锁，直到所有共享锁都被释放。

全文 >>

Zk学习笔记——ZkClient

2020-07-11

tonglin0325

ZkClient是开源的zk客户端，对Zookeeper原生的java api进行了封装，实现了诸如session超时重连，watcher反复注册等功能。

依赖的话有

<dependency>
    <groupId>com.101tec</groupId>
    <artifactId>zkclient</artifactId>
    <version>0.10</version>
</dependency>

和

<dependency>
    <groupId>com.github.sgroschupf</groupId>
    <artifactId>zkclient</artifactId>
    <version>0.1</version>
</dependency>

前者使用较为广泛，使用其的项目包括kafka，dubbo等

1.创建，获取和删除节点

全文 >>

Zk学习笔记——修改节点

2020-07-11

tonglin0325

参考：从Paxos到Zookeeper分布式一致性原理和实践

使用的zk依赖是cdh5.16.2的3.4.5

<!-- zookeeper -->
<dependency>
    <groupId>org.apache.zookeeper</groupId>
    <artifactId>zookeeper</artifactId>
    <version>3.4.5-cdh5.16.2</version>
</dependency>

代码，创建一个临时节点，并赋值，此时初始的节点version=0

使用version=-1对这个节点进行修改，然后查看这个stat，version=1，

使用version=1对这个节点进行修改，成功，然后查看这个stat，version=2，

再次使用version=1对这个节点进行修改，失败，抛出BadVersionException

package com.bigdata.zookeeper;


import org.apache.zookeeper.*;
import org.apache.zookeeper.data.Stat;

import java.util.List;
import java.util.concurrent.CountDownLatch;

public class ZkExample implements Watcher {

    public static CountDownLatch connectedSemaphore = new CountDownLatch(1);
    private static ZooKeeper zk;

    public static void main(String[] args) throws Exception {
        zk = new ZooKeeper("master:2181", 5000, new ZkExample());
        System.out.println(zk.getState());
        try {
            connectedSemaphore.await();
            // 创建一个节点
            String path = "/app6";
            zk.create(path, "123".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
            zk.getData(path, true, null);

            // version=-1，修改节点
            Stat stat = zk.setData(path, "555".getBytes(), -1);
            System.out.println(stat.getVersion());

            // version=1，修改节点
            Stat stat2 = zk.setData(path, "55555".getBytes(), stat.getVersion());
            System.out.println(stat2.getVersion());

            // version=1，再次修改节点
            Stat stat3 = zk.setData(path, "555555555".getBytes(), stat.getVersion());

            Thread.sleep(10000); // 10秒延时

        } catch (InterruptedException e) {
            System.out.println("Zk session established" + e);
        }
    }

    @Override
    public void process(WatchedEvent watchedEvent) {
        System.out.println(watchedEvent);
        if (Event.KeeperState.SyncConnected == watchedEvent.getState()) {
            if (Event.EventType.None == watchedEvent.getType() &amp;&amp; null == watchedEvent.getPath()) {
                connectedSemaphore.countDown();
            } else if (watchedEvent.getType() == Event.EventType.NodeChildrenChanged) {
                try {
                    System.out.println(zk.getChildren(watchedEvent.getPath(), true));
                } catch (Exception e) {

                }
            }

        }
    }
}

输出

全文 >>

Zk学习笔记——检测节点是否存在

2020-07-11

tonglin0325

参考：从Paxos到Zookeeper分布式一致性原理和实践

使用的zk依赖是cdh5.16.2的3.4.5

<!-- zookeeper -->
<dependency>
    <groupId>org.apache.zookeeper</groupId>
    <artifactId>zookeeper</artifactId>
    <version>3.4.5-cdh5.16.2</version>
</dependency>

代码，exist函数来检查节点是否存在，同时会注册一个watch

package com.bigdata.zookeeper;


import org.apache.zookeeper.*;
import org.apache.zookeeper.data.Stat;

import java.util.List;
import java.util.concurrent.CountDownLatch;

public class ZkExample implements Watcher {

    public static CountDownLatch connectedSemaphore = new CountDownLatch(1);
    //    private static Stat stat = new Stat();
    private static ZooKeeper zk;

    public static void main(String[] args) throws Exception {
        zk = new ZooKeeper("master:2181", 5000, new ZkExample());
        System.out.println(zk.getState());
        try {
            connectedSemaphore.await();
            // 创建一个节点
            String path = "/app6";
            // 注册watch
            Stat stat = zk.exists(path, true);

            zk.create(path, "123".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);

            // version=-1，修改节点
            zk.setData(path, "555".getBytes(), -1);
            // 注册watch
            zk.exists(path, true);
            System.out.println(stat.getVersion());

            // version=1，修改节点
            Stat stat2 = zk.setData(path, "55555".getBytes(), stat.getVersion());
            // 注册watch
            zk.exists(path, true);
            System.out.println(stat2.getVersion());
            
            // version=-1，删除节点，正常
            zk.delete(path, stat2.getVersion());


            Thread.sleep(10000); // 10秒延时

        } catch (InterruptedException e) {
            System.out.println("Zk session established" + e);
        }
    }

    @Override
    public void process(WatchedEvent watchedEvent) {
        System.out.println(watchedEvent);
        if (Event.KeeperState.SyncConnected == watchedEvent.getState()) {
            if (Event.EventType.None == watchedEvent.getType() &amp;&amp; null == watchedEvent.getPath()) {
                connectedSemaphore.countDown();
            } else if (watchedEvent.getType() == Event.EventType.NodeChildrenChanged ||
                    watchedEvent.getType() == Event.EventType.NodeDeleted || watchedEvent.getType() == Event.EventType.NodeDataChanged) {
                try {
                    System.out.println(zk.getChildren(watchedEvent.getPath(), true));
//
                } catch (Exception e) {
                    System.out.println(e);
                }
            }

        }
    }
}

输出

Pac代理文件

2020-07-09

tonglin0325

用于将本地特定规则的请求转发到某个ip port代理

mac系统需要编辑~/.$hadow$ocksX-NG/gfwli$t.js中的FindProxyForURL函数

var proxy = "PROXY 127.0.0.1:xxxx; DIRECT;";

var rules = [
  "||xxx.xxx.com",
];

/*
* This file is part of Adblock Plus <http://adblockplus.org/>,
* Copyright (C) 2006-2014 Eyeo GmbH
*
* Adblock Plus is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 3 as
* published by the Free Software Foundation.
*
* Adblock Plus is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with Adblock Plus.  If not, see <http://www.gnu.org/licenses/>.
*/

function createDict()
{
    var result = {};
    result.__proto__ = null;
    return result;
}

function getOwnPropertyDescriptor(obj, key)
{
    if (obj.hasOwnProperty(key))
    {
        return obj[key];
    }
    return null;
}

function extend(subclass, superclass, definition)
{
    if (Object.__proto__)
    {
        definition.__proto__ = superclass.prototype;
        subclass.prototype = definition;
    }
    else
    {
        var tmpclass = function(){}, ret;
        tmpclass.prototype = superclass.prototype;
        subclass.prototype = new tmpclass();
        subclass.prototype.constructor = superclass;
        for (var i in definition)
        {
            if (definition.hasOwnProperty(i))
            {
                subclass.prototype[i] = definition[i];
            }
        }
    }
}

function Filter(text)
{
    this.text = text;
    this.subscriptions = [];
}
Filter.prototype = {
    text: null,
    subscriptions: null,
    toString: function()
    {
        return this.text;
    }
};
Filter.knownFilters = createDict();
Filter.elemhideRegExp = /^([^\/\*\|\@"!]*?)#(\@)?(?:([\w\-]+|\*)((?:\([\w\-]+(?:[$^*]?=[^\(\)"]*)?\))*)|#([^{}]+))$/;
Filter.regexpRegExp = /^(@@)?\/.*\/(?:\$~?[\w\-]+(?:=[^,\s]+)?(?:,~?[\w\-]+(?:=[^,\s]+)?)*)?$/;
Filter.optionsRegExp = /\$(~?[\w\-]+(?:=[^,\s]+)?(?:,~?[\w\-]+(?:=[^,\s]+)?)*)$/;
Filter.fromText = function(text)
{
    if (text in Filter.knownFilters)
    {
        return Filter.knownFilters[text];
    }
    var ret;
    if (text[0] == "!")
    {
        ret = new CommentFilter(text);
    }
    else
    {
        ret = RegExpFilter.fromText(text);
    }
    Filter.knownFilters[ret.text] = ret;
    return ret;
};

function InvalidFilter(text, reason)
{
    Filter.call(this, text);
    this.reason = reason;
}
extend(InvalidFilter, Filter, {
    reason: null
});

function CommentFilter(text)
{
    Filter.call(this, text);
}
extend(CommentFilter, Filter, {
});

function ActiveFilter(text, domains)
{
    Filter.call(this, text);
    this.domainSource = domains;
}
extend(ActiveFilter, Filter, {
    domainSource: null,
    domainSeparator: null,
    ignoreTrailingDot: true,
    domainSourceIsUpperCase: false,
    getDomains: function()
    {
        var prop = getOwnPropertyDescriptor(this, "domains");
        if (prop)
        {
            return prop;
        }
        var domains = null;
        if (this.domainSource)
        {
            var source = this.domainSource;
            if (!this.domainSourceIsUpperCase)
            {
                source = source.toUpperCase();
            }
            var list = source.split(this.domainSeparator);
            if (list.length == 1 &amp;&amp; list[0][0] != "~")
            {
                domains = createDict();
                domains[""] = false;
                if (this.ignoreTrailingDot)
                {
                    list[0] = list[0].replace(/\.+$/, "");
                }
                domains[list[0]] = true;
            }
            else
            {
                var hasIncludes = false;
                for (var i = 0; i < list.length; i++)
                {
                    var domain = list[i];
                    if (this.ignoreTrailingDot)
                    {
                        domain = domain.replace(/\.+$/, "");
                    }
                    if (domain == "")
                    {
                        continue;
                    }
                    var include;
                    if (domain[0] == "~")
                    {
                        include = false;
                        domain = domain.substr(1);
                    }
                    else
                    {
                        include = true;
                        hasIncludes = true;
                    }
                    if (!domains)
                    {
                        domains = createDict();
                    }
                    domains[domain] = include;
                }
                domains[""] = !hasIncludes;
            }
            this.domainSource = null;
        }
        return this.domains;
    },
    sitekeys: null,
    isActiveOnDomain: function(docDomain, sitekey)
    {
        if (this.getSitekeys() &amp;&amp; (!sitekey || this.getSitekeys().indexOf(sitekey.toUpperCase()) < 0))
        {
            return false;
        }
        if (!this.getDomains())
        {
            return true;
        }
        if (!docDomain)
        {
            return this.getDomains()[""];
        }
        if (this.ignoreTrailingDot)
        {
            docDomain = docDomain.replace(/\.+$/, "");
        }
        docDomain = docDomain.toUpperCase();
        while (true)
        {
            if (docDomain in this.getDomains())
            {
                return this.domains[docDomain];
            }
            var nextDot = docDomain.indexOf(".");
            if (nextDot < 0)
            {
                break;
            }
            docDomain = docDomain.substr(nextDot + 1);
        }
        return this.domains[""];
    },
    isActiveOnlyOnDomain: function(docDomain)
    {
        if (!docDomain || !this.getDomains() || this.getDomains()[""])
        {
            return false;
        }
        if (this.ignoreTrailingDot)
        {
            docDomain = docDomain.replace(/\.+$/, "");
        }
        docDomain = docDomain.toUpperCase();
        for (var domain in this.getDomains())
        {
            if (this.domains[domain] &amp;&amp; domain != docDomain &amp;&amp; (domain.length <= docDomain.length || domain.indexOf("." + docDomain) != domain.length - docDomain.length - 1))
            {
                return false;
            }
        }
        return true;
    }
});

function RegExpFilter(text, regexpSource, contentType, matchCase, domains, thirdParty, sitekeys)
{
    ActiveFilter.call(this, text, domains, sitekeys);
    if (contentType != null)
    {
        this.contentType = contentType;
    }
    if (matchCase)
    {
        this.matchCase = matchCase;
    }
    if (thirdParty != null)
    {
        this.thirdParty = thirdParty;
    }
    if (sitekeys != null)
    {
        this.sitekeySource = sitekeys;
    }
    if (regexpSource.length >= 2 &amp;&amp; regexpSource[0] == "/" &amp;&amp; regexpSource[regexpSource.length - 1] == "/")
    {
        var regexp = new RegExp(regexpSource.substr(1, regexpSource.length - 2), this.matchCase ? "" : "i");
        this.regexp = regexp;
    }
    else
    {
        this.regexpSource = regexpSource;
    }
}
extend(RegExpFilter, ActiveFilter, {
    domainSourceIsUpperCase: true,
    length: 1,
    domainSeparator: "|",
    regexpSource: null,
    getRegexp: function()
    {
        var prop = getOwnPropertyDescriptor(this, "regexp");
        if (prop)
        {
            return prop;
        }
        var source = this.regexpSource.replace(/\*+/g, "*").replace(/\^\|$/, "^").replace(/\W/g, "\\$&amp;").replace(/\\\*/g, ".*").replace(/\\\^/g, "(?:[\\x00-\\x24\\x26-\\x2C\\x2F\\x3A-\\x40\\x5B-\\x5E\\x60\\x7B-\\x7F]|$)").replace(/^\\\|\\\|/, "^[\\w\\-]+:\\/+(?!\\/)(?:[^\\/]+\\.)?").replace(/^\\\|/, "^").replace(/\\\|$/, "$").replace(/^(\.\*)/, "").replace(/(\.\*)$/, "");
        var regexp = new RegExp(source, this.matchCase ? "" : "i");
        this.regexp = regexp;
        return regexp;
    },
    contentType: 2147483647,
    matchCase: false,
    thirdParty: null,
    sitekeySource: null,
    getSitekeys: function()
    {
        var prop = getOwnPropertyDescriptor(this, "sitekeys");
        if (prop)
        {
            return prop;
        }
        var sitekeys = null;
        if (this.sitekeySource)
        {
            sitekeys = this.sitekeySource.split("|");
            this.sitekeySource = null;
        }
        this.sitekeys = sitekeys;
        return this.sitekeys;
    },
    matches: function(location, contentType, docDomain, thirdParty, sitekey)
    {
        if (this.getRegexp().test(location) &amp;&amp; this.isActiveOnDomain(docDomain, sitekey))
        {
            return true;
        }
        return false;
    }
});
RegExpFilter.prototype["0"] = "#this";
RegExpFilter.fromText = function(text)
{
    var blocking = true;
    var origText = text;
    if (text.indexOf("@@") == 0)
    {
        blocking = false;
        text = text.substr(2);
    }
    var contentType = null;
    var matchCase = null;
    var domains = null;
    var sitekeys = null;
    var thirdParty = null;
    var collapse = null;
    var options;
    var match = text.indexOf("$") >= 0 ? Filter.optionsRegExp.exec(text) : null;
    if (match)
    {
        options = match[1].toUpperCase().split(",");
        text = match.input.substr(0, match.index);
        for (var _loopIndex6 = 0; _loopIndex6 < options.length; ++_loopIndex6)
        {
            var option = options[_loopIndex6];
            var value = null;
            var separatorIndex = option.indexOf("=");
            if (separatorIndex >= 0)
            {
                value = option.substr(separatorIndex + 1);
                option = option.substr(0, separatorIndex);
            }
            option = option.replace(/-/, "_");
            if (option in RegExpFilter.typeMap)
            {
                if (contentType == null)
                {
                    contentType = 0;
                }
                contentType |= RegExpFilter.typeMap[option];
            }
            else if (option[0] == "~" &amp;&amp; option.substr(1) in RegExpFilter.typeMap)
            {
                if (contentType == null)
                {
                    contentType = RegExpFilter.prototype.contentType;
                }
                contentType &amp;= ~RegExpFilter.typeMap[option.substr(1)];
            }
            else if (option == "MATCH_CASE")
            {
                matchCase = true;
            }
            else if (option == "~MATCH_CASE")
            {
                matchCase = false;
            }
            else if (option == "DOMAIN" &amp;&amp; typeof value != "undefined")
            {
                domains = value;
            }
            else if (option == "THIRD_PARTY")
            {
                thirdParty = true;
            }
            else if (option == "~THIRD_PARTY")
            {
                thirdParty = false;
            }
            else if (option == "COLLAPSE")
            {
                collapse = true;
            }
            else if (option == "~COLLAPSE")
            {
                collapse = false;
            }
            else if (option == "SITEKEY" &amp;&amp; typeof value != "undefined")
            {
                sitekeys = value;
            }
            else
            {
                return new InvalidFilter(origText, "Unknown option " + option.toLowerCase());
            }
        }
    }
    if (!blocking &amp;&amp; (contentType == null || contentType &amp; RegExpFilter.typeMap.DOCUMENT) &amp;&amp; (!options || options.indexOf("DOCUMENT") < 0) &amp;&amp; !/^\|?[\w\-]+:/.test(text))
    {
        if (contentType == null)
        {
            contentType = RegExpFilter.prototype.contentType;
        }
        contentType &amp;= ~RegExpFilter.typeMap.DOCUMENT;
    }
    try
    {
        if (blocking)
        {
            return new BlockingFilter(origText, text, contentType, matchCase, domains, thirdParty, sitekeys, collapse);
        }
        else
        {
            return new WhitelistFilter(origText, text, contentType, matchCase, domains, thirdParty, sitekeys);
        }
    }
    catch (e)
    {
        return new InvalidFilter(origText, e);
    }
};
RegExpFilter.typeMap = {
    OTHER: 1,
    SCRIPT: 2,
    IMAGE: 4,
    STYLESHEET: 8,
    OBJECT: 16,
    SUBDOCUMENT: 32,
    DOCUMENT: 64,
    XBL: 1,
    PING: 1,
    XMLHTTPREQUEST: 2048,
    OBJECT_SUBREQUEST: 4096,
    DTD: 1,
    MEDIA: 16384,
    FONT: 32768,
    BACKGROUND: 4,
    POPUP: 268435456,
    ELEMHIDE: 1073741824
};
RegExpFilter.prototype.contentType &amp;= ~ (RegExpFilter.typeMap.ELEMHIDE | RegExpFilter.typeMap.POPUP);

function BlockingFilter(text, regexpSource, contentType, matchCase, domains, thirdParty, sitekeys, collapse)
{
    RegExpFilter.call(this, text, regexpSource, contentType, matchCase, domains, thirdParty, sitekeys);
    this.collapse = collapse;
}
extend(BlockingFilter, RegExpFilter, {
    collapse: null
});

function WhitelistFilter(text, regexpSource, contentType, matchCase, domains, thirdParty, sitekeys)
{
    RegExpFilter.call(this, text, regexpSource, contentType, matchCase, domains, thirdParty, sitekeys);
}
extend(WhitelistFilter, RegExpFilter, {
});

function Matcher()
{
    this.clear();
}
Matcher.prototype = {
    filterByKeyword: null,
    keywordByFilter: null,
    clear: function()
    {
        this.filterByKeyword = createDict();
        this.keywordByFilter = createDict();
    },
    add: function(filter)
    {
        if (filter.text in this.keywordByFilter)
        {
            return;
        }
        var keyword = this.findKeyword(filter);
        var oldEntry = this.filterByKeyword[keyword];
        if (typeof oldEntry == "undefined")
        {
            this.filterByKeyword[keyword] = filter;
        }
        else if (oldEntry.length == 1)
        {
            this.filterByKeyword[keyword] = [oldEntry, filter];
        }
        else
        {
            oldEntry.push(filter);
        }
        this.keywordByFilter[filter.text] = keyword;
    },
    remove: function(filter)
    {
        if (!(filter.text in this.keywordByFilter))
        {
            return;
        }
        var keyword = this.keywordByFilter[filter.text];
        var list = this.filterByKeyword[keyword];
        if (list.length <= 1)
        {
            delete this.filterByKeyword[keyword];
        }
        else
        {
            var index = list.indexOf(filter);
            if (index >= 0)
            {
                list.splice(index, 1);
                if (list.length == 1)
                {
                    this.filterByKeyword[keyword] = list[0];
                }
            }
        }
        delete this.keywordByFilter[filter.text];
    },
    findKeyword: function(filter)
    {
        var result = "";
        var text = filter.text;
        if (Filter.regexpRegExp.test(text))
        {
            return result;
        }
        var match = Filter.optionsRegExp.exec(text);
        if (match)
        {
            text = match.input.substr(0, match.index);
        }
        if (text.substr(0, 2) == "@@")
        {
            text = text.substr(2);
        }
        var candidates = text.toLowerCase().match(/[^a-z0-9%*][a-z0-9%]{3,}(?=[^a-z0-9%*])/g);
        if (!candidates)
        {
            return result;
        }
        var hash = this.filterByKeyword;
        var resultCount = 16777215;
        var resultLength = 0;
        for (var i = 0, l = candidates.length; i < l; i++)
        {
            var candidate = candidates[i].substr(1);
            var count = candidate in hash ? hash[candidate].length : 0;
            if (count < resultCount || count == resultCount &amp;&amp; candidate.length > resultLength)
            {
                result = candidate;
                resultCount = count;
                resultLength = candidate.length;
            }
        }
        return result;
    },
    hasFilter: function(filter)
    {
        return filter.text in this.keywordByFilter;
    },
    getKeywordForFilter: function(filter)
    {
        if (filter.text in this.keywordByFilter)
        {
            return this.keywordByFilter[filter.text];
        }
        else
        {
            return null;
        }
    },
    _checkEntryMatch: function(keyword, location, contentType, docDomain, thirdParty, sitekey)
    {
        var list = this.filterByKeyword[keyword];
        for (var i = 0; i < list.length; i++)
        {
            var filter = list[i];
            if (filter == "#this")
            {
                filter = list;
            }
            if (filter.matches(location, contentType, docDomain, thirdParty, sitekey))
            {
                return filter;
            }
        }
        return null;
    },
    matchesAny: function(location, contentType, docDomain, thirdParty, sitekey)
    {
        var candidates = location.toLowerCase().match(/[a-z0-9%]{3,}/g);
        if (candidates === null)
        {
            candidates = [];
        }
        candidates.push("");
        for (var i = 0, l = candidates.length; i < l; i++)
        {
            var substr = candidates[i];
            if (substr in this.filterByKeyword)
            {
                var result = this._checkEntryMatch(substr, location, contentType, docDomain, thirdParty, sitekey);
                if (result)
                {
                    return result;
                }
            }
        }
        return null;
    }
};

function CombinedMatcher()
{
    this.blacklist = new Matcher();
    this.whitelist = new Matcher();
    this.resultCache = createDict();
}
CombinedMatcher.maxCacheEntries = 1000;
CombinedMatcher.prototype = {
    blacklist: null,
    whitelist: null,
    resultCache: null,
    cacheEntries: 0,
    clear: function()
    {
        this.blacklist.clear();
        this.whitelist.clear();
        this.resultCache = createDict();
        this.cacheEntries = 0;
    },
    add: function(filter)
    {
        if (filter instanceof WhitelistFilter)
        {
            this.whitelist.add(filter);
        }
        else
        {
            this.blacklist.add(filter);
        }
        if (this.cacheEntries > 0)
        {
            this.resultCache = createDict();
            this.cacheEntries = 0;
        }
    },
    remove: function(filter)
    {
        if (filter instanceof WhitelistFilter)
        {
            this.whitelist.remove(filter);
        }
        else
        {
            this.blacklist.remove(filter);
        }
        if (this.cacheEntries > 0)
        {
            this.resultCache = createDict();
            this.cacheEntries = 0;
        }
    },
    findKeyword: function(filter)
    {
        if (filter instanceof WhitelistFilter)
        {
            return this.whitelist.findKeyword(filter);
        }
        else
        {
            return this.blacklist.findKeyword(filter);
        }
    },
    hasFilter: function(filter)
    {
        if (filter instanceof WhitelistFilter)
        {
            return this.whitelist.hasFilter(filter);
        }
        else
        {
            return this.blacklist.hasFilter(filter);
        }
    },
    getKeywordForFilter: function(filter)
    {
        if (filter instanceof WhitelistFilter)
        {
            return this.whitelist.getKeywordForFilter(filter);
        }
        else
        {
            return this.blacklist.getKeywordForFilter(filter);
        }
    },
    isSlowFilter: function(filter)
    {
        var matcher = filter instanceof WhitelistFilter ? this.whitelist : this.blacklist;
        if (matcher.hasFilter(filter))
        {
            return !matcher.getKeywordForFilter(filter);
        }
        else
        {
            return !matcher.findKeyword(filter);
        }
    },
    matchesAnyInternal: function(location, contentType, docDomain, thirdParty, sitekey)
    {
        var candidates = location.toLowerCase().match(/[a-z0-9%]{3,}/g);
        if (candidates === null)
        {
            candidates = [];
        }
        candidates.push("");
        var blacklistHit = null;
        for (var i = 0, l = candidates.length; i < l; i++)
        {
            var substr = candidates[i];
            if (substr in this.whitelist.filterByKeyword)
            {
                var result = this.whitelist._checkEntryMatch(substr, location, contentType, docDomain, thirdParty, sitekey);
                if (result)
                {
                    return result;
                }
            }
            if (substr in this.blacklist.filterByKeyword &amp;&amp; blacklistHit === null)
            {
                blacklistHit = this.blacklist._checkEntryMatch(substr, location, contentType, docDomain, thirdParty, sitekey);
            }
        }
        return blacklistHit;
    },
    matchesAny: function(location, docDomain)
    {
        var key = location + " " + docDomain + " ";
        if (key in this.resultCache)
        {
            return this.resultCache[key];
        }
        var result = this.matchesAnyInternal(location, 0, docDomain, null, null);
        if (this.cacheEntries >= CombinedMatcher.maxCacheEntries)
        {
            this.resultCache = createDict();
            this.cacheEntries = 0;
        }
        this.resultCache[key] = result;
        this.cacheEntries++;
        return result;
    }
};
var defaultMatcher = new CombinedMatcher();

var direct = 'DIRECT;';

for (var i = 0; i < rules.length; i++) {
    defaultMatcher.add(Filter.fromText(rules[i]));
}

function FindProxyForURL(url, host) {
    if (defaultMatcher.matchesAny(url, host) instanceof BlockingFilter) {
        return proxy;
    }
    return direct;
}

Zk学习笔记——创建节点

2020-07-05

tonglin0325

参考：从Paxos到Zookeeper分布式一致性原理和实践

使用的zk依赖是cdh5.16.2的3.4.5

<!-- zookeeper -->
<dependency>
    <groupId>org.apache.zookeeper</groupId>
    <artifactId>zookeeper</artifactId>
    <version>3.4.5-cdh5.16.2</version>
</dependency>

Zk在创建节点的有同步创建和异步创建，但是Zk是不支持递归创建节点，即父节点必须存储。

在创建同名的节点的时候，会抛出NodeExistsException异常。

Zookeeper节点内容只支持字节数组（byte[]）

同步创建节点代码，其中创建了一个永久节点，key是/app1，value是123

package com.bigdata.zookeeper;


import org.apache.zookeeper.*;
import java.util.concurrent.CountDownLatch;

public class ZkExample implements Watcher {

    public static CountDownLatch connectedSemaphore = new CountDownLatch(1);

    public static void main(String[] args) throws Exception {
        ZooKeeper zk = new ZooKeeper("master:2181", 5000, new ZkExample());
        System.out.println(zk.getState());
        try {
            connectedSemaphore.await();
            String path = zk.create("/app1", "123".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);
            System.out.println("success create znode: " + path);
        } catch (InterruptedException e) {
            System.out.println("Zk session established" + e);
        }
    }

    @Override
    public void process(WatchedEvent watchedEvent) {
        System.out.println(watchedEvent);
        if (Event.KeeperState.SyncConnected == watchedEvent.getState()) {
            connectedSemaphore.countDown();
        }
    }
}

输出，使用Zkui查看zk中的数据，能发现多出了/app1的节点，节点的值是123

全文 >>

Zk学习笔记——读取节点

2020-07-05

tonglin0325

参考：从Paxos到Zookeeper分布式一致性原理和实践

使用的zk依赖是cdh5.16.2的3.4.5

<!-- zookeeper -->
<dependency>
    <groupId>org.apache.zookeeper</groupId>
    <artifactId>zookeeper</artifactId>
    <version>3.4.5-cdh5.16.2</version>
</dependency>

代码，在初始化zk的时候，会触发一个watchEvent，将CountDownLatch-1=0，从而开始读取/app1的value，此时为12345，

然后将其修改成123，会再次触发一个watchEvent，此时watchEvent的type是NodeDataChanged，之后10秒延时过后，输出/app1的value，此时为123

读取节点的getData放中的true表示是否注册一个watch，注意这里的watch是一次性的，触发通过之后就失效，需要反复注册watch

package com.bigdata.zookeeper;


import org.apache.zookeeper.AsyncCallback;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.data.Stat;

import java.util.concurrent.CountDownLatch;

public class ZkExample implements Watcher {

    public static CountDownLatch connectedSemaphore = new CountDownLatch(1);
    private static Stat stat = new Stat();

    public static void main(String[] args) throws Exception {
        ZooKeeper zk = new ZooKeeper("master:2181", 5000, new ZkExample());
        System.out.println(zk.getState());
        try {
            connectedSemaphore.await();
            System.out.println(new String(zk.getData("/app1", true, stat)));
            Thread.sleep(10000); // 10秒延时
            System.out.println(new String(zk.getData("/app1", true, stat)));
        } catch (InterruptedException e) {
            System.out.println("Zk session established" + e);
        }
    }

    @Override
    public void process(WatchedEvent watchedEvent) {
        System.out.println(watchedEvent);
        if (Event.KeeperState.SyncConnected == watchedEvent.getState()) {
            connectedSemaphore.countDown();
        }
    }
}

输出

异步代码

全文 >>

Zk学习笔记——连接Zookeeper

2020-07-05

tonglin0325

参考：从Paxos到Zookeeper分布式一致性原理和实践

使用的zk依赖是cdh5.16.2的3.4.5

<!-- zookeeper -->
<dependency>
    <groupId>org.apache.zookeeper</groupId>
    <artifactId>zookeeper</artifactId>
    <version>3.4.5-cdh5.16.2</version>
</dependency>

代码，其中CountDownLatch计数器参考：Java多线程——其他工具类CyclicBarrier、CountDownLatch和Exchange

初始化的计数器为1，当zk client连接到zk集群后，zk集群返回的状态为connected，计数器-1，触发InterruptedException

package com.bigdata.zookeeper;

import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;

import java.util.concurrent.CountDownLatch;

public class ZkExample implements Watcher {

    public static CountDownLatch connectedSemaphore = new CountDownLatch(1);

    public static void main(String[] args) throws Exception {
        ZooKeeper zk = new ZooKeeper("master:2181", 5000, new ZkExample());
        System.out.println(zk.getState());
        try {
            connectedSemaphore.await();
        } catch (InterruptedException e) {
            System.out.println("Zk session established");
        }
    }

    @Override
    public void process(WatchedEvent watchedEvent) {
        System.out.println(watchedEvent);
        if (Event.KeeperState.SyncConnected == watchedEvent.getState()) {
            connectedSemaphore.countDown();
        }
    }
}

输出

tonglin0325的个人主页

Kafka学习笔记——存储结构

分布式协议——Paxos、Raft和ZAB

Zk学习笔记——应用场景

Zk学习笔记——ZkClient

Zk学习笔记——修改节点

Zk学习笔记——检测节点是否存在

Pac代理文件

Zk学习笔记——创建节点

Zk学习笔记——读取节点

Zk学习笔记——连接Zookeeper

tonglin0325

标签

标签云

文章归档

最近文章