tonglin0325的个人主页

Zk学习笔记——权限控制

参考:从Paxos到Zookeeper分布式一致性原理和实践

使用的zk依赖是cdh5.16.2的3.4.5

1
2
3
4
5
6
7
<!-- zookeeper -->
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.4.5-cdh5.16.2</version>
</dependency>

Zookeeper提供了多种权限控制模式,分别是world,auth,digest,ip和super。

下面介绍模式scheme中的digest

使用如下语句对zk session添加权限,其中的username:password是账号密码

1
2
zk1.addAuthInfo("digest", "username:password".getBytes());

如果操作zk节点没有权限的话,会抛出NoAuthException

1
2
Exception in thread "main" org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /app6

代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
package com.bigdata.zookeeper;

import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.ZooDefs;
import org.apache.zookeeper.ZooKeeper;

public class AuthExample {

private static ZooKeeper zk1;
private static ZooKeeper zk2;

public static void main(String[] args) throws Exception {
// zk1 session
zk1 = new ZooKeeper("master:2181", 5000, null);
zk1.addAuthInfo("digest", "username:password".getBytes());
// 创建一个节点
String path = "/app6";
zk1.create(path, "123".getBytes(), ZooDefs.Ids.CREATOR_ALL_ACL, CreateMode.EPHEMERAL);

// zk2 session无权限
zk2 = new ZooKeeper("master:2181", 5000, null);
// System.out.println(new String(zk2.getData(path, false, null)));

// zk2 session有权限
zk2.addAuthInfo("digest", "username:password".getBytes());
System.out.println(new String(zk2.getData(path, false, null)));
}

}

没有权限的话,zkui也会报错

使用zookeeper-client访问

1
2
3
lintong@master:/opt/cloudera/parcels/CDH/bin$ ./zookeeper-client
Connecting to localhost:2181

查看,仍然没有权限

1
2
3
4
5
[zk: localhost:2181(CONNECTED) 0] ls /
[cluster, controller, brokers, zookeeper, admin, isr_change_notification, log_dir_event_notification, ngdata, controller_epoch, kafka-manager, solr, app6, consumers, hive_zookeeper_namespace_hive, latest_producer_id_block, app2, config, app1, hbase, app4, app3]
[zk: localhost:2181(CONNECTED) 1] ls /app6
Authentication is not valid : /app6

设置密码并查看

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[zk: localhost:2181(CONNECTED) 3] addauth digest username:password

[zk: localhost:2181(CONNECTED) 7] get /app6
123
cZxid = 0x139e88
ctime = Sun Aug 02 23:38:30 CST 2020
mZxid = 0x139e88
mtime = Sun Aug 02 23:38:30 CST 2020
pZxid = 0x139e88
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 3
numChildren = 0

全文 >>

guava学习笔记

1.CaseFormat

CaseFormat是guava中用于字符串格式转换的工具有以下几种类型

UPPER_CAMEL,比如 UpperCamel

UPPER_UNDERSCORE,比如 UPPER_UNDERSCORE

LOWER_CAMEL,比如 lowerCamel

LOWER_HYPHEN,比如 lower-hyphen

LOWER_UNDERSCORE,比如 lower_underscore

全文 >>

Elasticsearch学习笔记——别名

别名是一个指针或者名称,可以对应一个或者多个具体的索引。

别名的创建,这样就给一个名为es的索引添加了一个别名:alias_test

1
2
3
lintong@lintongdeMacBook-Pro ~ $ curl -XPUT 'http://master:9200/es/_alias/alias_test'
{"acknowledged":true}%

添加了别名之后,在查询es索引的时候

1
2
curl 'http://master:9200/alias_test/_search?q=_id:1'

就等同于

1
2
curl 'http://master:9200/es/_search?q=_id:1'

别名的删除

1
2
3
lintong@lintongdeMacBook-Pro ~ $ curl -XDELETE 'http://master:9200/es/_alias/alias_test'
{"acknowledged":true}%

全文 >>

Kafka学习笔记——存储结构

1,由cdh安装的kafka的默认存储路径如图所示在/var/local/kafka/data,一般会进行修改

kafka配置参考:apache kafka系列之server.properties配置文件参数说明

路径下文件如下

如果是多个路径的话,使用,进行分隔,比如/data01/kafka/data,/data02/kafka/data,注意data的权限需要是kafka用户和kafka组

对应kafka manager上的topic

具体的topic的目录下的文件

1
2
3
4
lintong@master:/var/local/kafka/data/test_topic-0$ ls
00000000000000000187.index 00000000000000000187.snapshot leader-epoch-checkpoint
00000000000000000187.log 00000000000000000187.timeindex

其中

187就是这个日志数据文件开始的offset

00000000000000000187.log是日志数据文件

可以使用解码命令查看日志片段中的内容

1
2
3
4
5
6
7
8
/opt/cloudera/parcels/KAFKA/bin/kafka-run-class kafka.tools.DumpLogSegments --files ./00000000000000000187.log
20/07/19 17:28:55 INFO utils.Log4jControllerRegistration$: Registered kafka:type=kafka.Log4jController MBean
Dumping ./00000000000000000187.log
Starting offset: 187
baseOffset: 187 lastOffset: 187 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false position: 0 CreateTime: 1595149737820 size: 71 magic: 2 compresscodec: NONE crc: 1738564593 isvalid: true
baseOffset: 188 lastOffset: 188 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false position: 71 CreateTime: 1595149744449 size: 74 magic: 2 compresscodec: NONE crc: 3794338178 isvalid: true
baseOffset: 189 lastOffset: 189 count: 1 baseSequence: -1 lastSequence: -1 producerId: -1 producerEpoch: -1 partitionLeaderEpoch: 0 isTransactional: false isControl: false position: 145 CreateTime: 1595149757514 size: 75 magic: 2 compresscodec: NONE crc: 3084259622 isvalid: true

log.segment.bytes设置是1G,如果log文件的大小达到1G之后会生成另外一个log文件

该参数在1.0.1及以下的kafka有bug,可能会影响消费者消费topic的数据,但是不影响生产者,参考:https://issues.apache.org/jira/browse/KAFKA-6292

当kafka的broker读取segment文件的时候,会判断当前当前读取的segment的偏移量position在继续读取一段HEADER_SIZE_UP_TO_MAGIC之后和该segment文件最大可读取的偏移量end之间的大小

当调高了log.segment.bytes=2G,注意此处2G=有符号INT的最大值=2147483647,有可能导致position+HEADER_SIZE_UP_TO_MAGIC的大小超过int最大值,从而成为负数,小于end,返回null,并导致从log文件的末尾开始读取数据

参考kafka 1.0.1版本源码:https://github.com/apache/kafka/blob/1.0.1/clients/src/main/java/org/apache/kafka/common/record/FileLogInputStream.java

全文 >>

分布式协议——Paxos、Raft和ZAB

参考:分布式系统协议Paxos、Raft和ZAB

Paxos算法是一种提高分布式系统容错率的一致性算法

Paxos 算法的步骤是这样:

1.首先有两种角色,一个是“提议者”,一个是“接受者”。提议者可以向接受者提出提议,然后接受者表达意见。

2.因为存在多个提议者,如果同时表达意见会出现意见不一致的情况,所以首先需要尽快选出一个领导者,让意见统一。

3.然后领导者会给接受者发出提议,如果一个提议被大多数接受者接纳,这个提议就通过了。

全文 >>

Zk学习笔记——应用场景

1.Master选举

在分布式系统中,需要选举一台机器作为master或者leader。

这时候,可以选择一个跟节点,比如/master,然后多台机器同时像这个节点创建一个子节点/master/lock,利用zookeeper的特性,最终只有一台机器能否创建成功,成功的那台机器就是Master;

其他机器注册watch到这个子节点,然后当master宕机的时候,其他机器就会重新开始选举。

全文 >>

Zk学习笔记——ZkClient

ZkClient是开源的zk客户端,对Zookeeper原生的java api进行了封装,实现了诸如session超时重连,watcher反复注册等功能。

依赖的话有

1
2
3
4
5
6
<dependency>
<groupId>com.101tec</groupId>
<artifactId>zkclient</artifactId>
<version>0.10</version>
</dependency>

1
2
3
4
5
6
<dependency>
<groupId>com.github.sgroschupf</groupId>
<artifactId>zkclient</artifactId>
<version>0.1</version>
</dependency>

前者使用较为广泛,使用其的项目包括kafka,dubbo等

全文 >>

Zk学习笔记——修改节点

参考:从Paxos到Zookeeper分布式一致性原理和实践

使用的zk依赖是cdh5.16.2的3.4.5

1
2
3
4
5
6
7
<!-- zookeeper -->
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.4.5-cdh5.16.2</version>
</dependency>

代码,创建一个临时节点,并赋值,此时初始的节点version=0

使用version=-1对这个节点进行修改,然后查看这个stat,version=1,

使用version=1对这个节点进行修改,成功,然后查看这个stat,version=2,

再次使用version=1对这个节点进行修改,失败,抛出BadVersionException

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
package com.bigdata.zookeeper;


import org.apache.zookeeper.*;
import org.apache.zookeeper.data.Stat;

import java.util.List;
import java.util.concurrent.CountDownLatch;

public class ZkExample implements Watcher {

public static CountDownLatch connectedSemaphore = new CountDownLatch(1);
private static ZooKeeper zk;

public static void main(String[] args) throws Exception {
zk = new ZooKeeper("master:2181", 5000, new ZkExample());
System.out.println(zk.getState());
try {
connectedSemaphore.await();
// 创建一个节点
String path = "/app6";
zk.create(path, "123".getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
zk.getData(path, true, null);

// version=-1,修改节点
Stat stat = zk.setData(path, "555".getBytes(), -1);
System.out.println(stat.getVersion());

// version=1,修改节点
Stat stat2 = zk.setData(path, "55555".getBytes(), stat.getVersion());
System.out.println(stat2.getVersion());

// version=1,再次修改节点
Stat stat3 = zk.setData(path, "555555555".getBytes(), stat.getVersion());

Thread.sleep(10000); // 10秒延时

} catch (InterruptedException e) {
System.out.println("Zk session established" + e);
}
}

@Override
public void process(WatchedEvent watchedEvent) {
System.out.println(watchedEvent);
if (Event.KeeperState.SyncConnected == watchedEvent.getState()) {
if (Event.EventType.None == watchedEvent.getType() &amp;&amp; null == watchedEvent.getPath()) {
connectedSemaphore.countDown();
} else if (watchedEvent.getType() == Event.EventType.NodeChildrenChanged) {
try {
System.out.println(zk.getChildren(watchedEvent.getPath(), true));
} catch (Exception e) {

}
}

}
}
}

输出

全文 >>

Pac代理文件

用于将本地特定规则的请求转发到某个ip port代理

mac系统需要编辑~/.$hadow$ocksX-NG/gfwli$t.js中的FindProxyForURL函数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
var proxy = "PROXY 127.0.0.1:xxxx; DIRECT;";

var rules = [
"||xxx.xxx.com",
];

/*
* This file is part of Adblock Plus <http://adblockplus.org/>,
* Copyright (C) 2006-2014 Eyeo GmbH
*
* Adblock Plus is free software: you can redistribute it and/or modify
* it under the terms of the GNU General Public License version 3 as
* published by the Free Software Foundation.
*
* Adblock Plus is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with Adblock Plus. If not, see <http://www.gnu.org/licenses/>.
*/

function createDict()
{
var result = {};
result.__proto__ = null;
return result;
}

function getOwnPropertyDescriptor(obj, key)
{
if (obj.hasOwnProperty(key))
{
return obj[key];
}
return null;
}

function extend(subclass, superclass, definition)
{
if (Object.__proto__)
{
definition.__proto__ = superclass.prototype;
subclass.prototype = definition;
}
else
{
var tmpclass = function(){}, ret;
tmpclass.prototype = superclass.prototype;
subclass.prototype = new tmpclass();
subclass.prototype.constructor = superclass;
for (var i in definition)
{
if (definition.hasOwnProperty(i))
{
subclass.prototype[i] = definition[i];
}
}
}
}

function Filter(text)
{
this.text = text;
this.subscriptions = [];
}
Filter.prototype = {
text: null,
subscriptions: null,
toString: function()
{
return this.text;
}
};
Filter.knownFilters = createDict();
Filter.elemhideRegExp = /^([^\/\*\|\@"!]*?)#(\@)?(?:([\w\-]+|\*)((?:\([\w\-]+(?:[$^*]?=[^\(\)"]*)?\))*)|#([^{}]+))$/;
Filter.regexpRegExp = /^(@@)?\/.*\/(?:\$~?[\w\-]+(?:=[^,\s]+)?(?:,~?[\w\-]+(?:=[^,\s]+)?)*)?$/;
Filter.optionsRegExp = /\$(~?[\w\-]+(?:=[^,\s]+)?(?:,~?[\w\-]+(?:=[^,\s]+)?)*)$/;
Filter.fromText = function(text)
{
if (text in Filter.knownFilters)
{
return Filter.knownFilters[text];
}
var ret;
if (text[0] == "!")
{
ret = new CommentFilter(text);
}
else
{
ret = RegExpFilter.fromText(text);
}
Filter.knownFilters[ret.text] = ret;
return ret;
};

function InvalidFilter(text, reason)
{
Filter.call(this, text);
this.reason = reason;
}
extend(InvalidFilter, Filter, {
reason: null
});

function CommentFilter(text)
{
Filter.call(this, text);
}
extend(CommentFilter, Filter, {
});

function ActiveFilter(text, domains)
{
Filter.call(this, text);
this.domainSource = domains;
}
extend(ActiveFilter, Filter, {
domainSource: null,
domainSeparator: null,
ignoreTrailingDot: true,
domainSourceIsUpperCase: false,
getDomains: function()
{
var prop = getOwnPropertyDescriptor(this, "domains");
if (prop)
{
return prop;
}
var domains = null;
if (this.domainSource)
{
var source = this.domainSource;
if (!this.domainSourceIsUpperCase)
{
source = source.toUpperCase();
}
var list = source.split(this.domainSeparator);
if (list.length == 1 &amp;&amp; list[0][0] != "~")
{
domains = createDict();
domains[""] = false;
if (this.ignoreTrailingDot)
{
list[0] = list[0].replace(/\.+$/, "");
}
domains[list[0]] = true;
}
else
{
var hasIncludes = false;
for (var i = 0; i < list.length; i++)
{
var domain = list[i];
if (this.ignoreTrailingDot)
{
domain = domain.replace(/\.+$/, "");
}
if (domain == "")
{
continue;
}
var include;
if (domain[0] == "~")
{
include = false;
domain = domain.substr(1);
}
else
{
include = true;
hasIncludes = true;
}
if (!domains)
{
domains = createDict();
}
domains[domain] = include;
}
domains[""] = !hasIncludes;
}
this.domainSource = null;
}
return this.domains;
},
sitekeys: null,
isActiveOnDomain: function(docDomain, sitekey)
{
if (this.getSitekeys() &amp;&amp; (!sitekey || this.getSitekeys().indexOf(sitekey.toUpperCase()) < 0))
{
return false;
}
if (!this.getDomains())
{
return true;
}
if (!docDomain)
{
return this.getDomains()[""];
}
if (this.ignoreTrailingDot)
{
docDomain = docDomain.replace(/\.+$/, "");
}
docDomain = docDomain.toUpperCase();
while (true)
{
if (docDomain in this.getDomains())
{
return this.domains[docDomain];
}
var nextDot = docDomain.indexOf(".");
if (nextDot < 0)
{
break;
}
docDomain = docDomain.substr(nextDot + 1);
}
return this.domains[""];
},
isActiveOnlyOnDomain: function(docDomain)
{
if (!docDomain || !this.getDomains() || this.getDomains()[""])
{
return false;
}
if (this.ignoreTrailingDot)
{
docDomain = docDomain.replace(/\.+$/, "");
}
docDomain = docDomain.toUpperCase();
for (var domain in this.getDomains())
{
if (this.domains[domain] &amp;&amp; domain != docDomain &amp;&amp; (domain.length <= docDomain.length || domain.indexOf("." + docDomain) != domain.length - docDomain.length - 1))
{
return false;
}
}
return true;
}
});

function RegExpFilter(text, regexpSource, contentType, matchCase, domains, thirdParty, sitekeys)
{
ActiveFilter.call(this, text, domains, sitekeys);
if (contentType != null)
{
this.contentType = contentType;
}
if (matchCase)
{
this.matchCase = matchCase;
}
if (thirdParty != null)
{
this.thirdParty = thirdParty;
}
if (sitekeys != null)
{
this.sitekeySource = sitekeys;
}
if (regexpSource.length >= 2 &amp;&amp; regexpSource[0] == "/" &amp;&amp; regexpSource[regexpSource.length - 1] == "/")
{
var regexp = new RegExp(regexpSource.substr(1, regexpSource.length - 2), this.matchCase ? "" : "i");
this.regexp = regexp;
}
else
{
this.regexpSource = regexpSource;
}
}
extend(RegExpFilter, ActiveFilter, {
domainSourceIsUpperCase: true,
length: 1,
domainSeparator: "|",
regexpSource: null,
getRegexp: function()
{
var prop = getOwnPropertyDescriptor(this, "regexp");
if (prop)
{
return prop;
}
var source = this.regexpSource.replace(/\*+/g, "*").replace(/\^\|$/, "^").replace(/\W/g, "\\$&amp;").replace(/\\\*/g, ".*").replace(/\\\^/g, "(?:[\\x00-\\x24\\x26-\\x2C\\x2F\\x3A-\\x40\\x5B-\\x5E\\x60\\x7B-\\x7F]|$)").replace(/^\\\|\\\|/, "^[\\w\\-]+:\\/+(?!\\/)(?:[^\\/]+\\.)?").replace(/^\\\|/, "^").replace(/\\\|$/, "$").replace(/^(\.\*)/, "").replace(/(\.\*)$/, "");
var regexp = new RegExp(source, this.matchCase ? "" : "i");
this.regexp = regexp;
return regexp;
},
contentType: 2147483647,
matchCase: false,
thirdParty: null,
sitekeySource: null,
getSitekeys: function()
{
var prop = getOwnPropertyDescriptor(this, "sitekeys");
if (prop)
{
return prop;
}
var sitekeys = null;
if (this.sitekeySource)
{
sitekeys = this.sitekeySource.split("|");
this.sitekeySource = null;
}
this.sitekeys = sitekeys;
return this.sitekeys;
},
matches: function(location, contentType, docDomain, thirdParty, sitekey)
{
if (this.getRegexp().test(location) &amp;&amp; this.isActiveOnDomain(docDomain, sitekey))
{
return true;
}
return false;
}
});
RegExpFilter.prototype["0"] = "#this";
RegExpFilter.fromText = function(text)
{
var blocking = true;
var origText = text;
if (text.indexOf("@@") == 0)
{
blocking = false;
text = text.substr(2);
}
var contentType = null;
var matchCase = null;
var domains = null;
var sitekeys = null;
var thirdParty = null;
var collapse = null;
var options;
var match = text.indexOf("$") >= 0 ? Filter.optionsRegExp.exec(text) : null;
if (match)
{
options = match[1].toUpperCase().split(",");
text = match.input.substr(0, match.index);
for (var _loopIndex6 = 0; _loopIndex6 < options.length; ++_loopIndex6)
{
var option = options[_loopIndex6];
var value = null;
var separatorIndex = option.indexOf("=");
if (separatorIndex >= 0)
{
value = option.substr(separatorIndex + 1);
option = option.substr(0, separatorIndex);
}
option = option.replace(/-/, "_");
if (option in RegExpFilter.typeMap)
{
if (contentType == null)
{
contentType = 0;
}
contentType |= RegExpFilter.typeMap[option];
}
else if (option[0] == "~" &amp;&amp; option.substr(1) in RegExpFilter.typeMap)
{
if (contentType == null)
{
contentType = RegExpFilter.prototype.contentType;
}
contentType &amp;= ~RegExpFilter.typeMap[option.substr(1)];
}
else if (option == "MATCH_CASE")
{
matchCase = true;
}
else if (option == "~MATCH_CASE")
{
matchCase = false;
}
else if (option == "DOMAIN" &amp;&amp; typeof value != "undefined")
{
domains = value;
}
else if (option == "THIRD_PARTY")
{
thirdParty = true;
}
else if (option == "~THIRD_PARTY")
{
thirdParty = false;
}
else if (option == "COLLAPSE")
{
collapse = true;
}
else if (option == "~COLLAPSE")
{
collapse = false;
}
else if (option == "SITEKEY" &amp;&amp; typeof value != "undefined")
{
sitekeys = value;
}
else
{
return new InvalidFilter(origText, "Unknown option " + option.toLowerCase());
}
}
}
if (!blocking &amp;&amp; (contentType == null || contentType &amp; RegExpFilter.typeMap.DOCUMENT) &amp;&amp; (!options || options.indexOf("DOCUMENT") < 0) &amp;&amp; !/^\|?[\w\-]+:/.test(text))
{
if (contentType == null)
{
contentType = RegExpFilter.prototype.contentType;
}
contentType &amp;= ~RegExpFilter.typeMap.DOCUMENT;
}
try
{
if (blocking)
{
return new BlockingFilter(origText, text, contentType, matchCase, domains, thirdParty, sitekeys, collapse);
}
else
{
return new WhitelistFilter(origText, text, contentType, matchCase, domains, thirdParty, sitekeys);
}
}
catch (e)
{
return new InvalidFilter(origText, e);
}
};
RegExpFilter.typeMap = {
OTHER: 1,
SCRIPT: 2,
IMAGE: 4,
STYLESHEET: 8,
OBJECT: 16,
SUBDOCUMENT: 32,
DOCUMENT: 64,
XBL: 1,
PING: 1,
XMLHTTPREQUEST: 2048,
OBJECT_SUBREQUEST: 4096,
DTD: 1,
MEDIA: 16384,
FONT: 32768,
BACKGROUND: 4,
POPUP: 268435456,
ELEMHIDE: 1073741824
};
RegExpFilter.prototype.contentType &amp;= ~ (RegExpFilter.typeMap.ELEMHIDE | RegExpFilter.typeMap.POPUP);

function BlockingFilter(text, regexpSource, contentType, matchCase, domains, thirdParty, sitekeys, collapse)
{
RegExpFilter.call(this, text, regexpSource, contentType, matchCase, domains, thirdParty, sitekeys);
this.collapse = collapse;
}
extend(BlockingFilter, RegExpFilter, {
collapse: null
});

function WhitelistFilter(text, regexpSource, contentType, matchCase, domains, thirdParty, sitekeys)
{
RegExpFilter.call(this, text, regexpSource, contentType, matchCase, domains, thirdParty, sitekeys);
}
extend(WhitelistFilter, RegExpFilter, {
});

function Matcher()
{
this.clear();
}
Matcher.prototype = {
filterByKeyword: null,
keywordByFilter: null,
clear: function()
{
this.filterByKeyword = createDict();
this.keywordByFilter = createDict();
},
add: function(filter)
{
if (filter.text in this.keywordByFilter)
{
return;
}
var keyword = this.findKeyword(filter);
var oldEntry = this.filterByKeyword[keyword];
if (typeof oldEntry == "undefined")
{
this.filterByKeyword[keyword] = filter;
}
else if (oldEntry.length == 1)
{
this.filterByKeyword[keyword] = [oldEntry, filter];
}
else
{
oldEntry.push(filter);
}
this.keywordByFilter[filter.text] = keyword;
},
remove: function(filter)
{
if (!(filter.text in this.keywordByFilter))
{
return;
}
var keyword = this.keywordByFilter[filter.text];
var list = this.filterByKeyword[keyword];
if (list.length <= 1)
{
delete this.filterByKeyword[keyword];
}
else
{
var index = list.indexOf(filter);
if (index >= 0)
{
list.splice(index, 1);
if (list.length == 1)
{
this.filterByKeyword[keyword] = list[0];
}
}
}
delete this.keywordByFilter[filter.text];
},
findKeyword: function(filter)
{
var result = "";
var text = filter.text;
if (Filter.regexpRegExp.test(text))
{
return result;
}
var match = Filter.optionsRegExp.exec(text);
if (match)
{
text = match.input.substr(0, match.index);
}
if (text.substr(0, 2) == "@@")
{
text = text.substr(2);
}
var candidates = text.toLowerCase().match(/[^a-z0-9%*][a-z0-9%]{3,}(?=[^a-z0-9%*])/g);
if (!candidates)
{
return result;
}
var hash = this.filterByKeyword;
var resultCount = 16777215;
var resultLength = 0;
for (var i = 0, l = candidates.length; i < l; i++)
{
var candidate = candidates[i].substr(1);
var count = candidate in hash ? hash[candidate].length : 0;
if (count < resultCount || count == resultCount &amp;&amp; candidate.length > resultLength)
{
result = candidate;
resultCount = count;
resultLength = candidate.length;
}
}
return result;
},
hasFilter: function(filter)
{
return filter.text in this.keywordByFilter;
},
getKeywordForFilter: function(filter)
{
if (filter.text in this.keywordByFilter)
{
return this.keywordByFilter[filter.text];
}
else
{
return null;
}
},
_checkEntryMatch: function(keyword, location, contentType, docDomain, thirdParty, sitekey)
{
var list = this.filterByKeyword[keyword];
for (var i = 0; i < list.length; i++)
{
var filter = list[i];
if (filter == "#this")
{
filter = list;
}
if (filter.matches(location, contentType, docDomain, thirdParty, sitekey))
{
return filter;
}
}
return null;
},
matchesAny: function(location, contentType, docDomain, thirdParty, sitekey)
{
var candidates = location.toLowerCase().match(/[a-z0-9%]{3,}/g);
if (candidates === null)
{
candidates = [];
}
candidates.push("");
for (var i = 0, l = candidates.length; i < l; i++)
{
var substr = candidates[i];
if (substr in this.filterByKeyword)
{
var result = this._checkEntryMatch(substr, location, contentType, docDomain, thirdParty, sitekey);
if (result)
{
return result;
}
}
}
return null;
}
};

function CombinedMatcher()
{
this.blacklist = new Matcher();
this.whitelist = new Matcher();
this.resultCache = createDict();
}
CombinedMatcher.maxCacheEntries = 1000;
CombinedMatcher.prototype = {
blacklist: null,
whitelist: null,
resultCache: null,
cacheEntries: 0,
clear: function()
{
this.blacklist.clear();
this.whitelist.clear();
this.resultCache = createDict();
this.cacheEntries = 0;
},
add: function(filter)
{
if (filter instanceof WhitelistFilter)
{
this.whitelist.add(filter);
}
else
{
this.blacklist.add(filter);
}
if (this.cacheEntries > 0)
{
this.resultCache = createDict();
this.cacheEntries = 0;
}
},
remove: function(filter)
{
if (filter instanceof WhitelistFilter)
{
this.whitelist.remove(filter);
}
else
{
this.blacklist.remove(filter);
}
if (this.cacheEntries > 0)
{
this.resultCache = createDict();
this.cacheEntries = 0;
}
},
findKeyword: function(filter)
{
if (filter instanceof WhitelistFilter)
{
return this.whitelist.findKeyword(filter);
}
else
{
return this.blacklist.findKeyword(filter);
}
},
hasFilter: function(filter)
{
if (filter instanceof WhitelistFilter)
{
return this.whitelist.hasFilter(filter);
}
else
{
return this.blacklist.hasFilter(filter);
}
},
getKeywordForFilter: function(filter)
{
if (filter instanceof WhitelistFilter)
{
return this.whitelist.getKeywordForFilter(filter);
}
else
{
return this.blacklist.getKeywordForFilter(filter);
}
},
isSlowFilter: function(filter)
{
var matcher = filter instanceof WhitelistFilter ? this.whitelist : this.blacklist;
if (matcher.hasFilter(filter))
{
return !matcher.getKeywordForFilter(filter);
}
else
{
return !matcher.findKeyword(filter);
}
},
matchesAnyInternal: function(location, contentType, docDomain, thirdParty, sitekey)
{
var candidates = location.toLowerCase().match(/[a-z0-9%]{3,}/g);
if (candidates === null)
{
candidates = [];
}
candidates.push("");
var blacklistHit = null;
for (var i = 0, l = candidates.length; i < l; i++)
{
var substr = candidates[i];
if (substr in this.whitelist.filterByKeyword)
{
var result = this.whitelist._checkEntryMatch(substr, location, contentType, docDomain, thirdParty, sitekey);
if (result)
{
return result;
}
}
if (substr in this.blacklist.filterByKeyword &amp;&amp; blacklistHit === null)
{
blacklistHit = this.blacklist._checkEntryMatch(substr, location, contentType, docDomain, thirdParty, sitekey);
}
}
return blacklistHit;
},
matchesAny: function(location, docDomain)
{
var key = location + " " + docDomain + " ";
if (key in this.resultCache)
{
return this.resultCache[key];
}
var result = this.matchesAnyInternal(location, 0, docDomain, null, null);
if (this.cacheEntries >= CombinedMatcher.maxCacheEntries)
{
this.resultCache = createDict();
this.cacheEntries = 0;
}
this.resultCache[key] = result;
this.cacheEntries++;
return result;
}
};
var defaultMatcher = new CombinedMatcher();

var direct = 'DIRECT;';

for (var i = 0; i < rules.length; i++) {
defaultMatcher.add(Filter.fromText(rules[i]));
}

function FindProxyForURL(url, host) {
if (defaultMatcher.matchesAny(url, host) instanceof BlockingFilter) {
return proxy;
}
return direct;
}

全文 >>