tonglin0325的个人主页

Java数据库——PreparedStatement接口

tonglin0325

PreparedStatement接口是Statement的子接口，属于预处理操作，与直接使用Statement不同的是，PreparedStatement在操作时，是先在数据表中准备好了一条SQL语句，但是此SQL语句的具体内容暂时不设置，而是之后再进行设置。

全文 >>

Avro学习笔记——avro-tools工具

tonglin0325

1.下载avro-tools.jar

1
https://archive.apache.org/dist/avro/avro-1.10.1/java/

avro-tools.jar常用命令：Working with Apache Avro files in Amazon S3　　

也可以查看help

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
java -jar ./avro-tools-1.10.1.jar help
Version 1.10.1 of Apache Avro
Copyright 2010-2015 The Apache Software Foundation

This product includes software developed at
The Apache Software Foundation (https://www.apache.org/).
----------------
Available tools:
    canonical  Converts an Avro Schema to its canonical form
          cat  Extracts samples from files
      compile  Generates Java code for the given schema.
       concat  Concatenates avro files without re-compressing.
        count  Counts the records in avro files or folders
  fingerprint  Returns the fingerprint for the schemas.
   fragtojson  Renders a binary-encoded Avro datum as JSON.
     fromjson  Reads JSON records and writes an Avro data file.
     fromtext  Imports a text file into an avro data file.
      getmeta  Prints out the metadata of an Avro data file.
    getschema  Prints out schema of an Avro data file.
          idl  Generates a JSON schema from an Avro IDL file
 idl2schemata  Extract JSON schemata of the types from an Avro IDL file
       induce  Induce schema/protocol from Java class/interface via reflection.
   jsontofrag  Renders a JSON-encoded Avro datum as binary.
       random  Creates a file with randomly generated instances of a schema.
      recodec  Alters the codec of a data file.
       repair  Recovers data from a corrupt Avro Data file
  rpcprotocol  Output the protocol of a RPC service
   rpcreceive  Opens an RPC Server and listens for one message.
      rpcsend  Sends a single RPC message.
       tether  Run a tethered mapreduce job.
       tojson  Dumps an Avro data file as JSON, record per line or pretty.
       totext  Converts an Avro data file to a text file.
     totrevni  Converts an Avro data file to a Trevni file.
  trevni_meta  Dumps a Trevni file's metadata as JSON.
trevni_random  Create a Trevni file filled with random instances of a schema.
trevni_tojson  Dumps a Trevni file as JSON.

2.查看avro文件的schema

1
2
java -jar ./avro-tools-1.10.1.jar getschema ./xxxx.avro

3.查看avro文件内容的json格式

1
2
java -jar ./avro-tools-1.10.1.jar tojson ./nova_ads_access_log-0-0008589084.avro | less

4.使用avro-tools编译java代码

编译avro IDL文件，参考

1
2
https://avro.apache.org/docs/current/gettingstartedjava.html
https://yanbin.blog/convert-apache-avro-to-parquet-format-in-java/

定义schema文件kst.avsc

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
{
  "namespace": "com.linkedin.haivvreo",
  "name": "test_serializer",
  "type": "record",
  "fields": [
    { "name":"string1", "type":"string" },
    { "name":"int1", "type":"int" },
    { "name":"tinyint1", "type":"int" },
    { "name":"smallint1", "type":"int" },
    { "name":"bigint1", "type":"long" },
    { "name":"boolean1", "type":"boolean" },
    { "name":"float1", "type":"float" },
    { "name":"double1", "type":"double" },
    { "name":"list1", "type":{"type":"array", "items":"string"} },
    { "name":"map1", "type":{"type":"map", "values":"int"} },
    { "name":"struct1", "type":{"type":"record", "name":"struct1_name", "fields": [
          { "name":"sInt", "type":"int" }, { "name":"sBoolean", "type":"boolean" }, { "name":"sString", "type":"string" } ] } },
    { "name":"union1", "type":["float", "boolean", "string"] },
    { "name":"enum1", "type":{"type":"enum", "name":"enum1_values", "symbols":["BLUE","RED", "GREEN"]} },
    { "name":"nullableint", "type":["int", "null"] },
    { "name":"bytes1", "type":"bytes" },
    { "name":"fixed1", "type":{"type":"fixed", "name":"threebytes", "size":3} }
  ] }

编译avro IDL文件

1
2
java -jar ./src/main/resources/avro-tools-1.10.1.jar compile schema ./src/main/avro/kst.avsc ./src/main/java

全文 >>

Ubuntu下的MySQL安装

tonglin0325

1.安装mysql-server

1
2
3
sudo apt-get update
sudo apt-get install mysql-server mysql-client

2.重新启动mysql服务

1
2
sudo service mysql restart

3.让apache支持mysql

1
2
sudo apt-get install libapache2-mod-auth-mysql

16.04使用下面命令

1
2
sudo apt-get install libmysqlclient-dev

4.登录mysql

1
2
mysql -u root -p

如果修改了配置文件my.cnf配置文件，需要重启数据库（修改方法在下面），重启数据库之前需要先重新载入apparmor配置文件，使用下面命令重新载入：

1
2
sudo /etc/init.d/apparmor restart

重新启动数据库

1
2
sudo /etc/init.d/mysql start

5.查看数据库的编码

1.查看MySQL数据库服务器和数据库MySQL字符集。

1
2
SHOW VARIABLES LIKE 'character_set_%';

全文 >>

元注解(Annotation)

tonglin0325

J2SE 5.0提供了很多新的特征。其中一个很重要的特征就是对元数据(Metadata)的支持。在J2SE 5.0中，这种元数据称为注解(Annotation)。

通过使用注解，程序开发人员可以在不改变原有逻辑的情况下，在源文件嵌入一些补充的信息。

全文 >>

go学习笔记——基本语法

tonglin0325

1.*和&的区别

& 是取地址符号 , 即取得某个变量的地址 , 如 &a

  • 是指针运算符 , 可以表示一个变量是指针类型 , 也可以表示一个指针变量所指向的存储单元 , 也就是这个地址所存储的值

参考：Go中*和&区别

println打印对象只能打印出其指针，需要使用fmt.Printf，如下

1
2
fmt.Printf("%+v\n", user)

参考：Golang 通过fmt包输出完整struct信息

2.defer

defer是Go语言提供的一种用于注册延迟调用的机制：让函数或语句可以在当前函数执行完毕后（包括通过return正常结束或者panic导致的异常结束）执行。

defer语句通常用于一些成对操作的场景：打开连接/关闭连接；加锁/释放锁；打开文件/关闭文件等。

defer在一些需要回收资源的场景非常有用，可以很方便地在函数结束前做一些清理操作。在打开资源语句的下一行，直接一句defer就可以在函数返回前关闭资源，可谓相当优雅。

1
2
3
f, _ := os.Open("defer.txt")
defer f.Close()

参考：Golang之轻松化解defer的温柔陷阱

3.日志库

1.sirupsen/logrus

1
2
go get -u github.com/sirupsen/logrus

文档

1
2
https://pkg.go.dev/github.com/sirupsen/logrus#section-readme

使用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
package main

import (
  "os"
  "github.com/sirupsen/logrus"
)

// Create a new instance of the logger. You can have any number of instances.
var log = logrus.New()

func main() {
  // The API for setting attributes is a little different than the package level
  // exported logger. See Godoc.
  log.Out = os.Stdout

  // You could set this to any `io.Writer` such as a file
  // file, err := os.OpenFile("logrus.log", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
  // if err == nil {
  //  log.Out = file
  // } else {
  //  log.Info("Failed to log to file, using default stderr")
  // }

  log.WithFields(logrus.Fields{
    "animal": "walrus",
    "size":   10,
  }).Info("A group of walrus emerges from the ocean")
}

2.rs/zerolog

1
2
go get -u github.com/rs/zerolog/log

使用

1
2
3
4
5
log.Logger = log.With().Caller().Logger()
log.Info().Msg("hello world")

// Output: {"level": "info", "message": "hello world", "caller": "/go/src/your_project/some_file:21"}

3.uber/zap

1
2
go get -u go.uber.org/zap

文档

1
2
https://pkg.go.dev/go.uber.org/zap

zap提供了2种logger，分别是Logger和SugaredLogger

在性能要求高但是不是很重要的场景下，适合使用SugaredLogger

1
2
3
4
5
6
7
8
9
10
11
logger, _ := zap.NewProduction()
defer logger.Sync() // flushes buffer, if any
sugar := logger.Sugar()
sugar.Infow("failed to fetch URL",
  // Structured context as loosely typed key-value pairs.
  "url", url,
  "attempt", 3,
  "backoff", time.Second,
)
sugar.Infof("Failed to fetch URL: %s", url)

输出

1
2
3
{"level":"info","ts":1703922949.209576,"caller":"server/main.go:109","msg":"failed to fetch URL","url":"http://example.com","attempt":3,"backoff":1}
{"level":"info","ts":1703922949.209731,"caller":"server/main.go:115","msg":"Failed to fetch URL: http://example.com"}

在性能要求高且需要类型安全的场景下，适合使用Logger

1
2
3
4
5
6
7
8
9
10
logger, _ := zap.NewProduction()
defer logger.Sync()
url := "http://example.com"
logger.Info("failed to fetch URL",
	// Structured context as strongly typed Field values.
	zap.String("url", url),
	zap.Int("attempt", 3),
	zap.Duration("backoff", time.Second),
)

输出

1
2
{"level":"info","ts":1703923022.603034,"caller":"server/main.go:108","msg":"failed to fetch URL","url":"http://example.com","attempt":3,"backoff":1}

NewExample/NewDevelopment/NewProduction区别

NewExample适用于测试代码，它将DebugLevel及以上级别的日志以JSON格式标准输出，但省略了时间戳和调用函数，以保持示例输出简短和确定。

NewDevelopment适用于开发环境，它以人类友好的格式将DebugLevel及以上级别的日志写入标准错误。

NewProduction适用于生产环境，它将info level及以上级别的日志以JSON格式写入标准错误。

其他参考文档：Go 每日一库之 zap

golang常用库包：log日志记录-uber的Go日志库zap使用详解

4.kratos logrus

参考：go各框架的log日志

5.日志文件回滚

日志文件按时间回滚：natefinch/lumberjack

1
2
go get gopkg.in/natefinch/lumberjack.v2

4.strconv包

int转string

1
2
s := strconv.Itoa(i)

int64转string

1
2
s := strconv.FormatInt(i, 10)

string转int

1
2
i, err := strconv.Atoi(s)

string转int64

1
2
i, err := strconv.ParseInt(s, 10, 64)

float转string

1
2
3
4
v := 3.1415926535
s1 := strconv.FormatFloat(v, 'E', -1, 32)//float32
s2 := strconv.FormatFloat(v, 'E', -1, 64)//float64

string转float

1
2
3
4
s := "3.1415926535"
v1, err := strconv.ParseFloat(v, 32)
v2, err := strconv.ParseFloat(v, 64)

参考：Go语言从入门到精通 - 【精华篇】strconv包详解

5.属性复制

可以使用 jinzhu/copier

使用jinzhu/copier

1
2
go get github.com/jinzhu/copier

copy

1
2
copier.Copy(&employee, &user)

deepcopy，区别是deepcopy的时候，对dst的属性进行修改，是肯定不会影响src的

1
2
3
var dst User
copier.CopyWithOption(&dst, src, copier.Option{DeepCopy: true})

如果想在复制的时候，对属性进行修改，可以使用方法赋值，注意需要使用copier.Copy，如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
type User struct {
  Name string
  Age  int
}

func (u *User) DoubleAge() int {
  return 2 * u.Age
}

type Employee struct {
  Name      string
  DoubleAge int
  Role      string
}

func main() {
  user := User{Name: "dj", Age: 18}
  employee := Employee{}

  copier.Copy(&employee, &user)
  fmt.Printf("%#v\n", employee)
}

参考：Go 每日一库之 copier

其他类似的库还有ulule/deepcoper，参考：golang struct拷贝工具（类似于java中 BeanUtils.copyProperties()）

以及

1
2
3
4
5
6
7
8
9
10
11
12
13
https://github.com/jinzhu/copier
https://github.com/mohae/deepcopy
https://github.com/ulule/deepcopier
https://github.com/mitchellh/copystructure
https://github.com/globusdigital/deep-copy
https://github.com/getlantern/deepcopy
https://github.com/antlabs/deepcopy 
https://github.com/go-toolsmith/astcopy
https://github.com/qdm12/reprint
https://github.com/huandu/go-clone
https://github.com/wzshiming/deepclone 
https://github.com/davidwalter0/go-clone

6.其他数据结构

golang没有set，priorityqueue这些数据结构，可以使用emirpasic/gods

参考：https://github.com/emirpasic/gods

7.切片处理

可以使用samber/lo来对切片、数组或集合进行处理

1
2
go get github.com/samber/lo@v1

比如filter操作

1
2
3
4
5
even := lo.Filter([]int{1, 2, 3, 4}, func(x int, index int) bool {
    return x%2 == 0
})
// []int{2, 4}

map操作

1
2
3
4
5
6
7
import "github.com/samber/lo"

lo.Map([]int64{1, 2, 3, 4}, func(x int64, index int) string {
    return strconv.FormatInt(x, 10)
})
// []string{"1", "2", "3", "4"}

去重操作

1
2
3
uniqValues := lo.Uniq([]int{1, 2, 2, 1})
// []int{1, 2}

获得map的key，并转换成数组

不使用lo

1
2
3
4
5
6
7
8
9
m := map[string]int{
	"foo": 1,
	"bar": 2,
}
keys := make([]string, 0, len(m))
for key := range m {
	keys = append(keys, key)
}

使用lo

1
2
3
4
5
6
7
8
9
m := map[string]int{
	"foo": 1,
	"bar": 2,
}
keys := lo.Keys[string, int](m)
fmt.Println(keys)

// [foo bar]

range操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
result := lo.Range(4)
// [0, 1, 2, 3]

result := lo.Range(-4)
// [0, -1, -2, -3]

result := lo.RangeFrom(1, 5)
// [1, 2, 3, 4, 5]

result := lo.RangeFrom[float64](1.0, 5)
// [1.0, 2.0, 3.0, 4.0, 5.0]

result := lo.RangeWithSteps(0, 20, 5)
// [0, 5, 10, 15]

result := lo.RangeWithSteps[float32](-1.0, -4.0, -1.0)
// [-1.0, -2.0, -3.0]

result := lo.RangeWithSteps(1, 4, -1)
// []

result := lo.Range(0)
// []

8.时间处理

可以使用time或者carbon

1.time

字符串转time.Time

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
import (
	"fmt"
	"time"
)

func main() {
	timeStr := "2023-07-21 14:30:00"
	// DateTime   = "2006-01-02 15:04:05"
	// DateOnly   = "2006-01-02"
	// TimeOnly   = "15:04:05"
	// RFC3339     = "2006-01-02T15:04:05Z07:00"

	parsedTime, err := time.Parse(time.DateTime, timeStr)
	if err != nil {
		fmt.Println("解析时间字符串时出错:", err)
		return
	}

	fmt.Println("解析后的时间:", parsedTime)
}

time.Time转字符串

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import (
	"fmt"
	"time"
)

func main() {
	// 获取当前时间
	currentTime := time.Now()
	// DateTime   = "2006-01-02 15:04:05"
	// DateOnly   = "2006-01-02"
	// TimeOnly   = "15:04:05"
	// RFC3339     = "2006-01-02T15:04:05Z07:00"
	timeStr := currentTime.Format(time.DateOnly)
	// 输出格式化后的时间字符串
	fmt.Println("格式化后的时间字符串:", timeStr)
}

2.carbon

1
2
go get -u github.com/golang-module/carbon/v2

文档

1
2
https://pkg.go.dev/github.com/golang-module/carbon/v2

9.断言

可以使用stretchr/testify

1
2
go get -u github.com/stretchr/testify

使用assert

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import (
  "testing"
  "github.com/stretchr/testify/assert"
)

func TestSomething(t *testing.T) {

  // assert equality
  assert.Equal(t, 123, 123, "they should be equal")

  // assert inequality
  assert.NotEqual(t, 123, 456, "they should not be equal")

  // assert for nil (good for errors)
  assert.Nil(t, object)

  // assert for not nil (good when you expect something)
  if assert.NotNil(t, object) {

    // now we know that object isn't nil, we are safe to make
    // further assertions without causing any errors
    assert.Equal(t, "Something", object.Value)

  }

}

10.resty

在go中请求接口可以使用go-resty/resty框架

1
2
go get github.com/go-resty/resty/v2

get请求

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// Create a Resty Client
client := resty.New()

resp, err := client.R().
      SetQueryParams(map[string]string{
          "page_no": "1",
          "limit": "20",
          "sort":"name",
          "order": "asc",
          "random":strconv.FormatInt(time.Now().Unix(), 10),
      }).
      SetHeader("Accept", "application/json").
      SetResult(&Result{}).
      SetAuthToken("BC594900518B4F7EAC75BD37F019E08FBC594900518B4F7EAC75BD37F019E08F").
      Get("/search_result")

post请求

1
2
3
4
5
6
7
8
9
10
11
// Create a Resty Client
client := resty.New()

// POST JSON string
// No need to set content type, if you have client level setting
resp, err := client.R().
      SetHeader("Content-Type", "application/json").
      SetBody(`{"username":"testuser", "password":"testpass"}`).
      SetResult(&AuthSuccess{}).    // or SetResult(AuthSuccess{}).
      Post("https://myapp.com/login")

11.单元测试

使用testing框架进行单元测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
package dao

import (
	"fmt"
	"gin-template/internal/database"
	"gin-template/internal/model"
	"testing"
)

func Test_userDo_Create(t *testing.T) {
	user := model.User{
		Username: "test",
		Email:    "test@test",
	}
	SetDefault(database.DB)

	err := User.Create(&user)
	if err != nil {
		fmt.Println("creat user")
	}
}

如果遇到go testing flag provided but not defined: -test.v的报错，解决方法是添加一个init.go文件

1
2
3
4
5
6
7
8
package test

import "testing"

func init() {
	testing.Init()
}

然后在使用了flag的地方添加的import中

1
2
_ "gin-template/internal/test"

参考：问题记录：flag provided but not defined: -test.v 异常处理过程

12.打桩工具——go monkey

go monkey可以用于在单元测试中进行打桩（指补齐未实现的代码）

参考：Go单测从零到溜系列4—使用monkey打桩

全文 >>

Java反射机制

tonglin0325

如果要通过一个对象找到一个类的名称，此时就需要用到反射机制（反射技术是用来做框架的，一般情况下Java私有对象不能被访问，但是暴力反射可以访问私有对象）。

任何一个类如果没有明确地声明继承自哪个父类的时候，则默认继承Object类，所以getClass()方法是Object类中的。

文件在包java_reflect目录下

全文 >>

tonglin0325

标签

标签云

AWS Airbnb Android CDH Doris ELK Flink Git Grafana HAProxy HBase Hadoop Hive Hudi InfluxDB JDBC JMeter JVM Java JavaWeb LaTex Linkedin Linux ML ML Infra MPP Maven MySQL Nexus OpenTSDB Paper Play Prometheus Python React Redis RocksDB Scala Solr Spark SpringBoot SpringMVC Thrift YARN antlr arthas avro cassandra clickhouse confluent docker filebeat flume golang google gradle hexo impala jenkins k8s kafka kerberos kudu ldap mac mongo mybatis nginx nlp open-falcon openwrt parquet presto ranger scyllaDB twitter zookeeper 刷题 前端 图存储及计算 多线程 广告系统 开发工具 数据结构 杂谈 正则表达式 算法 系统设计 计算机网络 设计模式

文章归档

最近文章