tonglin0325的个人主页

Nginx修改时间戳

tonglin0325

1.安装nginx,注意不要安装nginx-full

1
2
3
4
sudo apt-get install nginx
sudo apt-get install nginx-common
sudo apt-get install nginx-extras

确认版本

1
2
3
4
5
6
7
8
apt list --installed | grep nginx

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

nginx/xenial-updates,xenial-updates,xenial-security,xenial-security,now 1.10.3-0ubuntu0.16.04.4 all [已安装]
nginx-common/xenial-updates,xenial-updates,xenial-security,xenial-security,now 1.10.3-0ubuntu0.16.04.4 all [已安装]
nginx-extras/xenial-updates,xenial-security,now 1.10.3-0ubuntu0.16.04.4 amd64 [已安装]

2.修改配置 /etc/nginx/nginx.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

#       access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;

server {
                listen       80;
                server_name  localhost;
                location / {
                    root   html;
                    index  index.html index.htm;
                }
            }

log_format access_json '{"ts":"$fmt_localtime",'
                           '"schema":"com.xxx.xxx",'
                           '"host":"$server_addr",'
                           '"clientip":"$remote_addr",'
                           '"size":$body_bytes_sent,'
                           '"responsetime":$request_time,'
                           '"upstreamtime":"$upstream_response_time",'
                           '"upstreamhost":"$upstream_addr",'
                           '"http_host":"$host",'
                           '"url":"$uri",'
                           '"domain":"$host",'
                           '"xff":"$http_x_forwarded_for",'
                           '"referer":"$http_referer",'
                           '"status":"$status"}';

access_log  /var/log/nginx/access.log  access_json;

map $host $fmt_localtime {
        default '';
    }

log_by_lua_block {
       ngx.var.fmt_localtime = ngx.localtime();
    }

}

3.输出的日志

1
2
3
{"ts":"2019-10-13 22:59:48","schema":"com.xxx.xxx","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-13 22:59:48","schema":"com.xxx.xxx","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}

全文 >>

Nginx打印json日志

tonglin0325

1.修改配置，在http{}中添加

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
log_format access_json '{"@timestamp":"$time_iso8601",'
                           '"host":"$server_addr",'
                           '"clientip":"$remote_addr",'
                           '"size":$body_bytes_sent,'
                           '"responsetime":$request_time,'
                           '"upstreamtime":"$upstream_response_time",'
                           '"upstreamhost":"$upstream_addr",'
                           '"http_host":"$host",'
                           '"url":"$uri",'
                           '"domain":"$host",'
                           '"xff":"$http_x_forwarded_for",'
                           '"referer":"$http_referer",'
                           '"status":"$status"}';
access_log  /var/log/nginx/access.log  access_json;

全文 >>

Ubuntu16.04安装nginx

tonglin0325

1.安装

1
2
sudo apt-get install nginx

2.启动

1
2
systemctl start nginx.service

如果和apache2的80端口冲突了，修改一下apache2的port

1
2
sudo vim /etc/apache2/ports.conf

冲突的话，日志/var/log/nginx/error.log中将会报

1
2
2019/10/12 14:25:31 [emerg] 23836#23836: listen() to 0.0.0.0:80, backlog 511 failed (98: Address already in use)

修改成8080

全文 >>

Hive学习笔记——parser

tonglin0325

Hive是如何解析SQL的呢,首先拿hive的建表语句来举例,比如下面的建表语句

1
2
create table test(id int,name string)row format delimited fields terminated by '\t';

然后使用hive的show create table语句来查看创建的表结构，这是一张text表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
CREATE TABLE `test`(
  `id` int, 
  `name` string)
ROW FORMAT SERDE 
  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
WITH SERDEPROPERTIES ( 
  'field.delim'='\t', 
  'serialization.format'='\t') 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.mapred.TextInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  'hdfs://master:8020/user/hive/warehouse/test'
TBLPROPERTIES (
  'transient_lastDdlTime'='1568561230')

当然还有其他各种建表语句，比如

csv表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
CREATE EXTERNAL TABLE `default.test_1`(
	  `key` string COMMENT 'from deserializer', 
	  `value` string COMMENT 'from deserializer')
	ROW FORMAT SERDE 
	  'org.apache.hadoop.hive.serde2.OpenCSVSerde' 
	WITH SERDEPROPERTIES ( 
	  'escapeChar'='\\', 
	  'quoteChar'='\'', 
	  'separatorChar'='\t') 
	STORED AS INPUTFORMAT 
	  'org.apache.hadoop.mapred.TextInputFormat' 
	OUTPUTFORMAT 
	  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
	LOCATION
	  'hdfs://master:8020/user/hive/warehouse/test'
	TBLPROPERTIES (
	  'COLUMN_STATS_ACCURATE'='false', 
	  'numFiles'='0', 
	  'numRows'='-1', 
	  'rawDataSize'='-1', 
	  'totalSize'='0', 
	  'transient_lastDdlTime'='xxxx')

全文 >>

Hive学习笔记——metadata

tonglin0325

Hive结构体系

1
2
https://blog.csdn.net/zhoudaxia/article/details/8855937

依赖

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.6.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-exec</artifactId>
    <version>1.1.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-metastore</artifactId>
    <version>1.1.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-jdbc</artifactId>
    <version>1.1.0</version>
</dependency>

有2种方法可以取得hive的元数据

**1.使用hive的jdbc接口**的getMetaData方法来获取hive表的相关元信息

1
2
3
4
5
6
7
import java.sql.*;

Class.forName("org.apache.hive.jdbc.HiveDriver");
Connection connection = DriverManager.getConnection("jdbc:hive2://xxxxx:10000", "xxxx", "xxxx");
Statement statement = connection.createStatement();
DatabaseMetaData meta = connection.getMetaData();

参考

1
2
https://blog.csdn.net/u010368839/article/details/76358831

hive metadata源码解析可以参考

1
2
https://cloud.tencent.com/developer/article/1330250

hive thrift接口可以参考

全文 >>

Ubuntu16.04安装Filebeat

tonglin0325

Filebeat官方文档地址

1
2
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html

下载和安装

1
2
3
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.3.1-linux-x86_64.tar.gz
tar xzvf filebeat-7.3.1-linux-x86_64.tar.gz

编写filebeat.yml

全文 >>

Ubuntu16.04安装Consul

tonglin0325

1.下载安装包

1
2
3
https://www.consul.io/downloads.html
wget https://releases.hashicorp.com/consul/1.5.3/consul_1.5.3_linux_amd64.zip

2.解压

1
2
unzip consul_1.5.3_linux_amd64.zip

3.mv

1
2
sudo mv consul /usr/local/bin/consul

4.启动

参考：https://blog.csdn.net/u010046908/article/details/61916389

-dev 开发模式启动的时候，数据是存储在内存中，重启之后数据将丢失

1
2
consul agent -dev

-server 生成模式启动的时候，如果是server的话需要指定-server，如果是client的话，需要指定-client，比如

1
2
consul agent -ui -server -bootstrap-expect 1 -data-dir /tmp/consul -node=consul-server -bind=192.168.1.100 -client=192.168.1.100

-bootstrap-expect 1 通知consul server我们现在准备加入的server节点个数，该参数是为了延迟日志复制的启动直到我们指定数量的server节点成功的加入后启动

-data-dir /tmp/consul 数据持久的路径

-node=consul-server 指定节点在集群中的名称

-bind=192.168.1.100 该地址用来在集群内部的通讯，集群内的所有节点到地址都必须是可达的，默认是0.0.0.0，这意味着Consulo会使用第一个可用的私有IP地址，Consul可以使用TCP和UDP并且可以使用共同的端口，如果存在防火墙，这两者协议必须是允许的

-client 指定节点为client，指定客户端接口的绑定地址，包括：HTTP、DNS、RPC，默认是127.0.0.1，只允许回环接口访问，也就是本机访问，如果要想同一局域网内的其他机器访问，需要修改成自己的内网ip

server节点，指定client等于内网ip，统一局域网的机器可以访问，指定client=0.0.0.0，外网机器可以访问

1
2
nohup ./consul agent -ui -server -bootstrap-expect 1 -data-dir /home/worker/projects/consul-1.5.3/consul-data -node=xxx -client=xxx >> ./logs/consul.log 2>&amp;1 &amp;

client节点，不指定client的话，只能本机访问client节点，指定client=0.0.0.0，外网机器可以访问

1
2
nohup ./consul agent -ui -data-dir /home/worker/projects/consul-1.5.3/consul-data -node=xxx -bind=xxxx -client=xxx >> ./logs/consul.log 2>&amp;1 &amp;

join

1
2
./consul join server的ip

当3台分布式部署的时候，需要如下部署

参考：https://blog.csdn.net/chenchong08/article/details/77885989

1
2
3
4
5
6
nohup ./consul agent -ui -server -bootstrap-expect 3 -data-dir /home/worker/projects/consul-1.5.3/consul-data -node=host1 -client=0.0.0.0 -bind=ip1 >> ./logs/consul.log 2>&amp;1 &amp;
nohup ./consul agent -ui -server -bootstrap-expect 3 -data-dir /home/worker/projects/consul-1.5.3/consul-data -node=host2 -client=0.0.0.0 -bind=ip2 >> ./logs/consul.log 2>&amp;1 &amp;
./consul join host1
nohup ./consul agent -ui -server -bootstrap-expect 3 -data-dir /home/worker/projects/consul-1.5.3/consul-data -node=host3 -client=0.0.0.0 -bind=ip3 >> ./logs/consul.log 2>&amp;1 &amp;
./consul join host2

全文 >>

tonglin0325

标签

标签云

AWS Airbnb Android CDH Doris ELK Flink Git Grafana HAProxy HBase Hadoop Hive Hudi InfluxDB JDBC JMeter JVM Java JavaWeb LaTex Linkedin Linux ML ML Infra MPP Maven MySQL Nexus OpenTSDB Paper Play Prometheus Python React Redis RocksDB Scala Solr Spark SpringBoot SpringMVC Thrift YARN antlr arthas avro cassandra clickhouse confluent docker filebeat flume golang google gradle hexo impala jenkins k8s kafka kerberos kudu ldap mac mongo mybatis nginx nlp open-falcon openwrt parquet presto ranger scyllaDB twitter zookeeper 刷题 前端 图存储及计算 多线程 广告系统 开发工具 数据结构 杂谈 正则表达式 算法 系统设计 计算机网络 设计模式

文章归档

最近文章