tonglin0325的个人主页

Nginx使用笔记

tonglin0325

1.请求转发

比如说我要将127.0.0.1/topics上的所有请求转发到xxx:xxx/上

修改 sudo vim /etc/nginx/nginx.conf

1
2
3
4
5
6
7
8
9
10
server {
       listen       80;
        server_name  127.0.0.1;
        location /topics {
            #root   html;
            #index  index.html index.htm;
        proxy_pass http://xxx:xxx;
        }
    }

2.代理前端和后端服务

3.Nginx使用本地缓存

4.代理

1.正向代理和反向代理

正向代理：代理服务器位于客户端和目标服务器之间，代表客户端向目标服务器发送请求。客户端需要配置代理服务器。

用途: 访问控制、内容过滤、匿名浏览、缓存。我们用来kexue上网的工具就属于正向代理。

反向代理：代理服务器位于客户端和目标服务器之间，代表服务器处理客户端请求。客户端不知道反向代理的存在。

用途: 负载均衡、安全防护、缓存、SSL加速。比如使用Nginx，HAProxy，Apache HTTP Server等作为反向代理。

2.透明代理和非透明代理

透明代理和非透明代理都是正向代理，位于客户端和目标服务器之间，代理客户端向目标服务器发送请求。

透明代理：网络管理员在网络边界部署透明代理来过滤不良内容。用户浏览网站时，代理自动拦截和检查内容，而用户并不知道代理的存在。

非透明代理：用户配置浏览器使用非透明代理，以便匿名访问互联网。用户在浏览器中手动设置代理服务器地址和端口。

5.X-Forwarded-For，X-Real-IP和Remote Address

X-Forwarded-For通常用于标识通过 HTTP 代理或负载均衡器的原始客户端 IP 地址。X-Forwarded-For 是一个 HTTP 扩展头部。HTTP/1.1（RFC 2616）协议并没有对它的定义，它最开始是由 Squid 这个缓存代理软件引入，用来表示 HTTP 请求端真实 IP。如今它已经成为事实上的标准，被各大 HTTP 代理、负载均衡等转发服务广泛使用，并被写入 RFC 7239（Forwarded HTTP Extension）标准之中。示例值： X-Forwarded-For: client1, proxy1, proxy2

**X-Real-IP **有些反向代理服务器（如 Nginx）会将原始客户端 IP 地址放在这个头字段中。X-Real-IP，这是一个自定义头部字段。X-Real-IP 通常被 HTTP 代理用来表示与它产生 TCP 连接的设备 IP，这个设备可能是其他代理，也可能是真正的请求端。需要注意的是，X-Real-IP 目前并不属于任何标准。示例值：X-Real-IP: 203.0.113.195

Remote Address 是指服务器端看到的客户端的IP地址，即发起请求的源 IP 地址。在网络通信中，remote address 通常用于识别和追踪请求的来源。在不同的代理和负载均衡场景中，remote address 的值可能会发生变化。

参考：聊聊HTTP的X-Forwarded-For 和 X-Real-IP

1.直接访问

如果客户端直接访问服务器而没有经过任何代理或负载均衡器，remote address 将是客户端的 IP 地址。例如，如果客户端的 IP 地址是 203.0.113.195，服务器将看到：

1
2
Remote Address: 203.0.113.195

2.经过正向代理

当请求经过正向代理（包括透明代理和非透明代理）时，remote address 可能会有所不同：

  • 透明代理: 客户端不知道代理的存在，remote address 仍然显示为客户端的 IP 地址。
  • 非透明代理: 客户端知道代理的存在并进行配置，remote address 显示为代理服务器的 IP 地址，而原始客户端的 IP 地址通常会添加到 X-Forwarded-For 头字段中（如果代理服务器配置了添加 X-Forwarded-For 头字段的话）。

经过非透明代理的请求：

1
2
3
Remote Address: 198.51.100.1 (代理服务器 IP)
X-Forwarded-For: 203.0.113.195 (原始客户端 IP)

全文 >>

使用filebeat发送nginx日志到kafka

tonglin0325

1.配置filebeat_nginx.yml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
filebeat.modules:
- module: nginx
  access:
    enabled: true
    var.paths: ["/var/log/nginx/access.log*"]
  error:
    enabled: true
    var.paths: ["/var/log/nginx/error.log*"]

#----------------------------------Kafka output--------------------------------#
output.kafka:
  version: "1.0.1"
  enabled: true
  hosts: ['xxx:9092', 'xxx:9092', 'xxx:9092']
  topic: 'temp'
  required_acks: 1  #default
  compression: gzip #default
  max_message_bytes: 1000000 #default
  codec.format:
    string: '%{[message]}'

2.启动filebeat

1
2
./filebeat -e -c filebeat_nginx.yml

3.访问nginx

1
2
tail -f /var/log/nginx/access.log

日志文件输出

1
2
3
4
5
6
{"ts":"2019-10-14 10:53:22","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:23","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:23","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:30","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:31","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}

kafka输出

1
2
3
4
5
6
{"ts":"2019-10-14 10:53:23","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:23","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:22","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:30","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-14 10:53:31","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}

全文 >>

由crt和key文件生成keystore文件

tonglin0325

该图转自知乎 海棠依旧

 

1.先生成p12文件,生成的时候需要指定密码

1
2
openssl pkcs12 -export -in your_crt.crt -inkey your_key.key -out your_p12.p12

2.再生成keystore文件

1
2
keytool -importkeystore -v  -srckeystore your_p12.p12 -srcstoretype pkcs12 -srcstorepass 你设置的密码 -destkeystore your_keystore.keystore -deststoretype jks -deststorepass 你设置的密码

3.然后就能使用这个keystore文件和密码来进行https的请求

 

参考

1
2
https://blog.csdn.net/u013944791/article/details/73551253

全文 >>

Nginx修改时间戳

tonglin0325

1.安装nginx,注意不要安装nginx-full

1
2
3
4
sudo apt-get install nginx
sudo apt-get install nginx-common
sudo apt-get install nginx-extras

确认版本

1
2
3
4
5
6
7
8
apt list --installed | grep nginx

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

nginx/xenial-updates,xenial-updates,xenial-security,xenial-security,now 1.10.3-0ubuntu0.16.04.4 all [已安装]
nginx-common/xenial-updates,xenial-updates,xenial-security,xenial-security,now 1.10.3-0ubuntu0.16.04.4 all [已安装]
nginx-extras/xenial-updates,xenial-security,now 1.10.3-0ubuntu0.16.04.4 amd64 [已安装]

2.修改配置 /etc/nginx/nginx.conf

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

#       access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;

server {
                listen       80;
                server_name  localhost;
                location / {
                    root   html;
                    index  index.html index.htm;
                }
            }

log_format access_json '{"ts":"$fmt_localtime",'
                           '"schema":"com.xxx.xxx",'
                           '"host":"$server_addr",'
                           '"clientip":"$remote_addr",'
                           '"size":$body_bytes_sent,'
                           '"responsetime":$request_time,'
                           '"upstreamtime":"$upstream_response_time",'
                           '"upstreamhost":"$upstream_addr",'
                           '"http_host":"$host",'
                           '"url":"$uri",'
                           '"domain":"$host",'
                           '"xff":"$http_x_forwarded_for",'
                           '"referer":"$http_referer",'
                           '"status":"$status"}';

access_log  /var/log/nginx/access.log  access_json;

map $host $fmt_localtime {
        default '';
    }

log_by_lua_block {
       ngx.var.fmt_localtime = ngx.localtime();
    }

}

3.输出的日志

1
2
3
{"ts":"2019-10-13 22:59:48","schema":"com.xxx.xxx","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}
{"ts":"2019-10-13 22:59:48","schema":"com.xxx.xxx","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}

 

Nginx打印json日志

tonglin0325

1.修改配置，在http{}中添加

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
log_format access_json '{"@timestamp":"$time_iso8601",'
                           '"host":"$server_addr",'
                           '"clientip":"$remote_addr",'
                           '"size":$body_bytes_sent,'
                           '"responsetime":$request_time,'
                           '"upstreamtime":"$upstream_response_time",'
                           '"upstreamhost":"$upstream_addr",'
                           '"http_host":"$host",'
                           '"url":"$uri",'
                           '"domain":"$host",'
                           '"xff":"$http_x_forwarded_for",'
                           '"referer":"$http_referer",'
                           '"status":"$status"}';
access_log  /var/log/nginx/access.log  access_json;

 

 

 2.重启

1
2
systemctl restart nginx

或者

1
2
nginx -s reload

3.访问，输出日志

1
2
{"@timestamp":"2019-10-12T18:41:48+08:00","host":"127.0.0.1","clientip":"127.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"localhost","url":"/index.html","domain":"localhost","xff":"-","referer":"-","status":"304"}

全文 >>

Nginx添加静态页面

tonglin0325

1.编辑配置文件

1
2
sudo vim  /etc/nginx/nginx.conf

在http {}中添加如下

1
2
3
4
5
6
7
8
9
server {
                listen       80;
                server_name  localhost;
                location / {
                    root   html;
                    index  index.html index.htm;
                }
            }

 

 

2.重启nginx

1
2
systemctl restart nginx

3.访问，成功

1
2
localhost:80/index.html

全文 >>

Ubuntu16.04安装nginx

tonglin0325

1.安装

1
2
sudo apt-get install nginx

2.启动

1
2
systemctl start nginx.service

如果和apache2的80端口冲突了，修改一下apache2的port

1
2
sudo vim /etc/apache2/ports.conf

冲突的话，日志/var/log/nginx/error.log中将会报

1
2
2019/10/12 14:25:31 [emerg] 23836#23836: listen() to 0.0.0.0:80, backlog 511 failed (98: Address already in use)

修改成8080

全文 >>

Nexus上传python包

tonglin0325

参考

1
2
https://blog.csdn.net/m0_37607365/article/details/79998955

1.首先创建pypi仓库

 

 

其中，PyPI类的服务，支持三种：     

proxy，提供代理服务

hosted，提供私有包的发布服务

全文 >>

Hive学习笔记——parser

tonglin0325

Hive是如何解析SQL的呢,首先拿hive的建表语句来举例,比如下面的建表语句

1
2
create table test(id int,name string)row format delimited fields terminated by '\t';

然后使用hive的show create table语句来查看创建的表结构，这是一张text表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
CREATE TABLE `test`(
  `id` int, 
  `name` string)
ROW FORMAT SERDE 
  'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' 
WITH SERDEPROPERTIES ( 
  'field.delim'='\t', 
  'serialization.format'='\t') 
STORED AS INPUTFORMAT 
  'org.apache.hadoop.mapred.TextInputFormat' 
OUTPUTFORMAT 
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
  'hdfs://master:8020/user/hive/warehouse/test'
TBLPROPERTIES (
  'transient_lastDdlTime'='1568561230')

当然还有其他各种建表语句，比如

csv表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
CREATE EXTERNAL TABLE `default.test_1`(
	  `key` string COMMENT 'from deserializer', 
	  `value` string COMMENT 'from deserializer')
	ROW FORMAT SERDE 
	  'org.apache.hadoop.hive.serde2.OpenCSVSerde' 
	WITH SERDEPROPERTIES ( 
	  'escapeChar'='\\', 
	  'quoteChar'='\'', 
	  'separatorChar'='\t') 
	STORED AS INPUTFORMAT 
	  'org.apache.hadoop.mapred.TextInputFormat' 
	OUTPUTFORMAT 
	  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
	LOCATION
	  'hdfs://master:8020/user/hive/warehouse/test'
	TBLPROPERTIES (
	  'COLUMN_STATS_ACCURATE'='false', 
	  'numFiles'='0', 
	  'numRows'='-1', 
	  'rawDataSize'='-1', 
	  'totalSize'='0', 
	  'transient_lastDdlTime'='xxxx')

 parquet表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
CREATE TABLE `default.test`(
	  `time` string, 
	  `server` int, 
	  `id` bigint)
	PARTITIONED BY ( 
	  `ds` string)
	ROW FORMAT SERDE 
	  'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe' 
	WITH SERDEPROPERTIES ( 
	  'field.delim'='\t', 
	  'serialization.format'='\t') 
	STORED AS INPUTFORMAT 
	  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat' 
	OUTPUTFORMAT 
	  'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
	LOCATION
	  'hdfs://master:8020/user/hive/warehouse/test'
	TBLPROPERTIES (
  'transient_lastDdlTime'='xxxx')

 json表

全文 >>

Ubuntu16.04安装Supervisor

tonglin0325

安装

1
2
sudo apt-get install supervisor

启动,否则会报 unix:///tmp/supervisor.sock no such file

1
2
service supervisor start

或者

1
2
supervisord -c /etc/supervisor/supervisord.conf

 生成配置文件

1
2
echo_supervisord_conf > /etc/supervisor/supervisord.conf

注意里面的注释去掉

1
2
3
4
5
[inet_http_server]         ; inet (TCP) server disabled by default
port=127.0.0.1:9001        ; (ip_address:port specifier, *:port for all iface)
username=user              ; (default is no username (open server))
password=123               ; (default is no password (open server))

全文 >>

tonglin0325

标签

标签云

AWS Airbnb Android CDH Doris ELK Flink Git Grafana HAProxy HBase Hadoop Hive Hudi InfluxDB JDBC JMeter JVM Java LaTex Linkedin Linux ML ML Infra MPP Maven MySQL Nexus OpenTSDB Paper Play Prometheus Python React Redis RocksDB Scala Solr Spark SpringBoot SpringMVC Thrift YARN antlr arthas avro cassandra clickhouse confluent docker filebeat flume golang google gradle hexo impala jenkins k8s kafka kerberos kudu ldap mac mongo mybatis nginx nlp open-falcon openwrt parquet presto ranger rocketmq scyllaDB twitter zookeeper 刷题 前端 图存储及计算 多线程 广告系统 开发工具 数据结构 杂谈 算法 系统设计 计算机基础 设计模式

文章归档

最近文章