tonglin0325的个人主页

Nginx修改时间戳

tonglin0325

1.安装nginx,注意不要安装nginx-full

1
2
3
4
sudo apt-get install nginx
sudo apt-get install nginx-common
sudo apt-get install nginx-extras

确认版本

1
2
3
4
5
6
7
8
apt list --installed | grep nginx

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

nginx/xenial-updates,xenial-updates,xenial-security,xenial-security,now 1.10.3-0ubuntu0.16.04.4 all [已安装]
nginx-common/xenial-updates,xenial-updates,xenial-security,xenial-security,now 1.10.3-0ubuntu0.16.04.4 all [已安装]
nginx-extras/xenial-updates,xenial-security,now 1.10.3-0ubuntu0.16.04.4 amd64 [已安装]

2.修改配置 /etc/nginx/nginx.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

#       access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;

server {
                listen       80;
                server_name  localhost;
                location / {
                    root   html;
                    index  index.html index.htm;
                }
            }

log_format access_json '{"ts":"$fmt_localtime",'
                           '"schema":"com.xxx.xxx",'
                           '"host":"$server_addr",'
                           '"clientip":"$remote_addr",'
                           '"size":$body_bytes_sent,'
                           '"responsetime":$request_time,'
                           '"upstreamtime":"$upstream_response_time",'
                           '"upstreamhost":"$upstream_addr",'
                           '"http_host":"$host",'
                           '"url":"$uri",'
                           '"domain":"$host",'
                           '"xff":"$http_x_forwarded_for",'
                           '"referer":"$http_referer",'
                           '"status":"$status"}';

access_log  /var/log/nginx/access.log  access_json;

map $host $fmt_localtime {
        default '';
    }

log_by_lua_block {
       ngx.var.fmt_localtime = ngx.localtime();
    }

}

3.输出的日志

1
2
3
{"ts":"2019-10-13 22:59:48","schema":"com.xxx.xxx","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-13 22:59:48","schema":"com.xxx.xxx","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}

全文 >>

Nginx打印json日志

tonglin0325

1.修改配置，在http{}中添加

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
log_format access_json '{"@timestamp":"$time_iso8601",'
                           '"host":"$server_addr",'
                           '"clientip":"$remote_addr",'
                           '"size":$body_bytes_sent,'
                           '"responsetime":$request_time,'
                           '"upstreamtime":"$upstream_response_time",'
                           '"upstreamhost":"$upstream_addr",'
                           '"http_host":"$host",'
                           '"url":"$uri",'
                           '"domain":"$host",'
                           '"xff":"$http_x_forwarded_for",'
                           '"referer":"$http_referer",'
                           '"status":"$status"}';
access_log  /var/log/nginx/access.log  access_json;

全文 >>

Ubuntu16.04安装nginx

tonglin0325

1.安装

1
2
sudo apt-get install nginx

2.启动

1
2
systemctl start nginx.service

如果和apache2的80端口冲突了，修改一下apache2的port

1
2
sudo vim /etc/apache2/ports.conf

冲突的话，日志/var/log/nginx/error.log中将会报

1
2
2019/10/12 14:25:31 [emerg] 23836#23836: listen() to 0.0.0.0:80, backlog 511 failed (98: Address already in use)

修改成8080

全文 >>

Hive学习笔记——parser

tonglin0325

Hive是如何解析SQL的呢,首先拿hive的建表语句来举例,比如下面的建表语句

1
2
create table test(id int,name string)row format delimited fields terminated by '\t';

然后使用hive的show create table语句来查看创建的表结构，这是一张text表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
CREATE TABLE `test`(
  `id` int, 
  `name` string)
ROW FORMAT SERDE 
  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
WITH SERDEPROPERTIES ( 
  'field.delim'='\t', 
  'serialization.format'='\t') 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.mapred.TextInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  'hdfs://master:8020/user/hive/warehouse/test'
TBLPROPERTIES (
  'transient_lastDdlTime'='1568561230')

当然还有其他各种建表语句，比如

csv表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
CREATE EXTERNAL TABLE `default.test_1`(
	  `key` string COMMENT 'from deserializer', 
	  `value` string COMMENT 'from deserializer')
	ROW FORMAT SERDE 
	  'org.apache.hadoop.hive.serde2.OpenCSVSerde' 
	WITH SERDEPROPERTIES ( 
	  'escapeChar'='\\', 
	  'quoteChar'='\'', 
	  'separatorChar'='\t') 
	STORED AS INPUTFORMAT 
	  'org.apache.hadoop.mapred.TextInputFormat' 
	OUTPUTFORMAT 
	  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
	LOCATION
	  'hdfs://master:8020/user/hive/warehouse/test'
	TBLPROPERTIES (
	  'COLUMN_STATS_ACCURATE'='false', 
	  'numFiles'='0', 
	  'numRows'='-1', 
	  'rawDataSize'='-1', 
	  'totalSize'='0', 
	  'transient_lastDdlTime'='xxxx')

全文 >>

Hive学习笔记——metadata

tonglin0325

Hive结构体系

1
2
https://blog.csdn.net/zhoudaxia/article/details/8855937

依赖

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.6.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-exec</artifactId>
    <version>1.1.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-metastore</artifactId>
    <version>1.1.0</version>
    <scope>provided</scope>
</dependency>
<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-jdbc</artifactId>
    <version>1.1.0</version>
</dependency>

有2种方法可以取得hive的元数据

**1.使用hive的jdbc接口**的getMetaData方法来获取hive表的相关元信息

1
2
3
4
5
6
7
import java.sql.*;

Class.forName("org.apache.hive.jdbc.HiveDriver");
Connection connection = DriverManager.getConnection("jdbc:hive2://xxxxx:10000", "xxxx", "xxxx");
Statement statement = connection.createStatement();
DatabaseMetaData meta = connection.getMetaData();

参考

1
2
https://blog.csdn.net/u010368839/article/details/76358831

hive metadata源码解析可以参考

1
2
https://cloud.tencent.com/developer/article/1330250

hive thrift接口可以参考

全文 >>

Ubuntu16.04安装Filebeat

tonglin0325

Filebeat官方文档地址

1
2
https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html

下载和安装

1
2
3
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.3.1-linux-x86_64.tar.gz
tar xzvf filebeat-7.3.1-linux-x86_64.tar.gz

编写filebeat.yml

全文 >>

tonglin0325

标签

标签云

AWS Airbnb Android CDH Doris ELK Flink Git Grafana HAProxy HBase Hadoop Hive Hudi InfluxDB JDBC JMeter JVM Java JavaWeb LaTex Linkedin Linux ML ML Infra MPP Maven MySQL Nexus OpenTSDB Paper Play Prometheus Python React Redis RocksDB Scala Solr Spark SpringBoot SpringMVC Thrift YARN antlr arthas avro cassandra clickhouse confluent docker filebeat flume golang google gradle hexo impala jenkins k8s kafka kerberos kudu ldap mac mongo mybatis nginx nlp open-falcon openwrt parquet presto ranger scyllaDB twitter zookeeper 刷题 前端 图存储及计算 多线程 广告系统 开发工具 数据结构 杂谈 算法 系统设计 计算机基础 设计模式

文章归档

最近文章