tonglin0325的个人主页

java使用gRPC框架

官方文档

1
2
https://grpc.io/docs/languages/java/quickstart/

官方example

1
2
https://github.com/grpc/grpc-java

1.定义proto文件#

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
syntax = "proto3";

option java_multiple_files = true;
option java_package = "io.grpc.examples.helloworld";
option java_outer_classname = "HelloWorldProto";
option objc_class_prefix = "HLW";

package helloworld;

// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
}

// The request message containing the user's name.
message HelloRequest {
string name = 1;
}

// The response message containing the greetings
message HelloReply {
string message = 1;
}

proto定义,参考官方文档

1
2
https://github.com/grpc/grpc-java/blob/master/examples/src/main/proto/helloworld.proto

2.编译proto文件#

1.使用protoc命令编译#

在项目目录下运行编译命令,里面使用了protoc-gen-grpc-java执行文件,需要参考:编译grpc-java项目生成protoc-gen-grpc-java文件

1
2
protoc --plugin=protoc-gen-grpc-java=./protoc-gen-grpc-java -I=./ --java_out=./src/main/java/ --grpc-java_out=./src/main/java/ ./src/main/proto/helloworld.proto

得到proto文件中定义的model和service的java代码

2.使用maven插件编译#

当然也可以使用maven插件来生成proto model和service的java代码,需要在pom.xml中添加插件,如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
<build>
<extensions>
<extension>
<groupId>kr.motd.maven</groupId>
<artifactId>os-maven-plugin</artifactId>
<version>1.5.0.Final</version>
</extension>
</extensions>
<plugins>
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
<version>0.5.1</version>
<configuration>
<protocArtifact>com.google.protobuf:protoc:3.5.1-1:exe:${os.detected.classifier}</protocArtifact>
<pluginId>grpc-java</pluginId>
<pluginArtifact>io.grpc:protoc-gen-grpc-java:1.14.0:exe:${os.detected.classifier}</pluginArtifact>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>compile-custom</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>8</source>
<target>8</target>
</configuration>
</plugin>

</plugins>
</build>

然后点击protobuf:compile和protobuf:comile-custome即可,参考:idea中.proto文件生成model类和service类

target目录下就会生成java代码

3.编写server和client#

1.添加依赖#

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
<!-- log4j -->
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<!--google-->
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>31.1-jre</version>
</dependency>
<!--grpc-->
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-netty-shaded</artifactId>
<version>1.59.0</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-protobuf</artifactId>
<version>1.59.0</version>
</dependency>
<dependency>
<groupId>io.grpc</groupId>
<artifactId>grpc-stub</artifactId>
<version>1.59.0</version>
</dependency>

2.server代码#

官方例子

1
2
https://github.com/grpc/grpc-java/blob/master/examples/src/main/java/io/grpc/examples/helloworld/HelloWorldServer.java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
package com.interview.rpc;

import io.grpc.Server;
import io.grpc.ServerBuilder;
import io.grpc.examples.helloworld.GreeterGrpc;
import io.grpc.examples.helloworld.HelloReply;
import io.grpc.examples.helloworld.HelloRequest;
import io.grpc.stub.StreamObserver;
import org.apache.log4j.Logger;

import java.io.IOException;

public class HelloWorldServer {

private static final Logger logger = Logger.getLogger(HelloWorldServer.class.getName());

private int port = 50051;
private Server server;

public static void main(String[] args) throws IOException, InterruptedException {
final HelloWorldServer server = new HelloWorldServer();
server.start();
server.blockUntilShutdown();
}

private void start() throws IOException {
server = ServerBuilder.forPort(port)
.addService(new GreeterImpl())
.build()
.start();
logger.info("Server started, listening on " + port);

Runtime.getRuntime().addShutdownHook(new Thread() {

@Override
public void run() {

System.err.println("*** shutting down gRPC server since JVM is shutting down");
HelloWorldServer.this.stop();
System.err.println("*** server shut down");
}
});
}

private void stop() {
if (server != null) {
server.shutdown();
}
}

// block 一直到退出程序
private void blockUntilShutdown() throws InterruptedException {
if (server != null) {
server.awaitTermination();
}
}

// 实现 定义一个实现服务接口的类
private class GreeterImpl extends GreeterGrpc.GreeterImplBase {
@Override
public void sayHello(HelloRequest req, StreamObserver<HelloReply> responseObserver) {
HelloReply reply = HelloReply.newBuilder().setMessage(("Hello " + req.getName())).build();
responseObserver.onNext(reply);
responseObserver.onCompleted();
System.out.println("Message from gRPC-Client:" + req.getName());
System.out.println("Message Response:" + reply.getMessage());
}
}

}

3.client代码#

官方例子

1
2
https://github.com/grpc/grpc-java/blob/master/examples/src/main/java/io/grpc/examples/helloworld/HelloWorldClient.java

代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
package com.interview.rpc;

import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import io.grpc.StatusRuntimeException;
import io.grpc.examples.helloworld.GreeterGrpc;
import io.grpc.examples.helloworld.HelloReply;
import io.grpc.examples.helloworld.HelloRequest;

import java.util.concurrent.TimeUnit;
import java.util.logging.Level;
import java.util.logging.Logger;

public class HelloWorldClient {

private final ManagedChannel channel;
private final GreeterGrpc.GreeterBlockingStub blockingStub;
private static final Logger logger = Logger.getLogger(HelloWorldClient.class.getName());

public static void main(String[] args) throws InterruptedException {
HelloWorldClient client = new HelloWorldClient("127.0.0.1", 50051);
try {
String user = "world";
if (args.length > 0) {
user = args[0];
}
client.greet(user);
} finally {
client.shutdown();
}
}

public HelloWorldClient(String host, int port) {
channel = ManagedChannelBuilder.forAddress(host, port)
.usePlaintext()
.build();

blockingStub = GreeterGrpc.newBlockingStub(channel);
}


public void shutdown() throws InterruptedException {
channel.shutdown().awaitTermination(5, TimeUnit.SECONDS);
}

public void greet(String name) {
HelloRequest request = HelloRequest.newBuilder().setName(name).build();
HelloReply response;
try {
response = blockingStub.sayHello(request);
} catch (StatusRuntimeException e) {
logger.log(Level.WARNING, "RPC failed: {0}", e.getStatus());
return;
}
logger.info("Message from gRPC-Server: " + response.getMessage());
}

}

4.运行server和client#

参考:【RPC基础系列3】gRPC简单示例

编译grpc-java项目生成protoc-gen-grpc-java文件

使用protoc命令生成service代码的时候,需要使用如下命令

1
2
protoc --plugin=protoc-gen-grpc-java=./protoc-gen-grpc-java -I=./ --java_out=./src/main/java/ --grpc-java_out=./src/main/java/ ./src/main/proto/helloworld.proto

其中的protoc-gen-grpc-java执行文件需要自己编译生成

编译的步骤如下

git clone grpc-java项目,

1
2
git clone git@github.com:grpc/grpc-java.git

切换到使用的grpc版本,比如v1.59.0,编译这个版本需要jdk11的支持

1
2
git checkout v1.59.0

使用gradlew命令进行编译

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
➜  /Users/lintong/coding/java/grpc-java/compiler git:(ae49d275b) $ ../gradlew java_pluginExecutable  -PskipAndroid=true
* Skipping the build of Android projects because skipAndroid=true

> Configure project :grpc-compiler
*** Building codegen requires Protobuf
*** Please refer to https://github.com/grpc/grpc-java/blob/master/COMPILING.md#how-to-build-code-generation-plugin

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.3/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 2s
2 actionable tasks: 1 executed, 1 up-to-date

在build目录下编译得到protoc-gen-grpc-java执行文件

1
2
3
4
➜  /Users/lintong/coding/java/grpc-java/compiler git:(ae49d275b) $ cd build/exe/java_plugin
➜ /Users/lintong/coding/java/grpc-java/compiler/build/exe/java_plugin git:(ae49d275b) $ ls
protoc-gen-grpc-java

这时候就可以使用protoc命令到编译proto文件得到service代码,否则只能得到model代码

官方文档:

1
2
https://github.com/grpc/grpc-java/tree/master/compiler

其他文档:java/go grpc 生成 service

全文 >>

go使用gRPC框架

官方文档

1
2
https://grpc.io/docs/languages/go/quickstart/

官方example

1
2
https://github.com/grpc/grpc-go

1.定义proto文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
syntax = "proto3";

option go_package = "google.golang.org/grpc/examples/helloworld/helloworld";
option java_multiple_files = true;
option java_package = "io.grpc.examples.helloworld";
option java_outer_classname = "HelloWorldProto";

package helloworld;

// The greeting service definition.
service Greeter {
// Sends a greeting
rpc SayHello (HelloRequest) returns (HelloReply) {}
}

// The request message containing the user's name.
message HelloRequest {
string name = 1;
}

// The response message containing the greetings
message HelloReply {
string message = 1;
}

proto定义,参考官方文档

1
2
https://github.com/grpc/grpc-go/blob/master/examples/helloworld/helloworld/helloworld.proto

2.编译proto文件

使用protoc命令编译

需要先安装compile插件,参考:https://grpc.io/docs/languages/go/quickstart/

1
2
3
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2

编译命令

1
2
protoc --go_out=./ --go_opt=paths=source_relative --go-grpc_out=./cmd/proto/helloworld --go-grpc_opt=paths=source_relative ./cmd/proto//helloworld/helloworld.proto

编译得到go代码

3.编写server和client代码

1.server代码

官方例子

1
2
https://github.com/grpc/grpc-go/blob/master/examples/helloworld/greeter_server/main.go

代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
package main

import (
"context"
"flag"
"fmt"
"log"
"net"

pb "awesome-project/cmd/proto/helloworld"
"google.golang.org/grpc"
)

var (
port = flag.Int("port", 50051, "The server port")
)

// server is used to implement helloworld.GreeterServer.
type server struct {
pb.UnimplementedGreeterServer
}

// SayHello implements helloworld.GreeterServer
func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {
log.Printf("Received: %v", in.GetName())
return &amp;pb.HelloReply{Message: "Hello " + in.GetName()}, nil
}

func main() {
flag.Parse()
lis, err := net.Listen("tcp", fmt.Sprintf(":%d", *port))
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
pb.RegisterGreeterServer(s, &amp;server{})
log.Printf("server listening at %v", lis.Addr())
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}

2.client代码

官方例子

1
2
https://github.com/grpc/grpc-go/blob/master/examples/helloworld/greeter_client/main.go

代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
package main

import (
"context"
"flag"
"log"
"time"

pb "awesome-project/cmd/proto/helloworld"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)

const (
defaultName = "world"
)

var (
addr = flag.String("addr", "localhost:50051", "the address to connect to")
name = flag.String("name", defaultName, "Name to greet")
)

func main() {
flag.Parse()
// Set up a connection to the server.
conn, err := grpc.Dial(*addr, grpc.WithTransportCredentials(insecure.NewCredentials()))
if err != nil {
log.Fatalf("did not connect: %v", err)
}
defer conn.Close()
c := pb.NewGreeterClient(conn)

// Contact the server and print out its response.
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
r, err := c.SayHello(ctx, &amp;pb.HelloRequest{Name: *name})
if err != nil {
log.Fatalf("could not greet: %v", err)
}
log.Printf("Greeting: %s", r.GetMessage())
}

3.运行server和client

成功看到hello world

全文 >>

go学习笔记——text template

golang可以使用text/template来实现模板生成文本,官方文档:https://pkg.go.dev/text/template

1.变量

可以在模板中定义变量,然后将这些变量赋值到模板的变量中

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import "text/template"

// 定义结构体
type Inventory struct {
Material string
Count uint
}
// 赋值
sweaters := Inventory{"wool", 17}
// 定义模板
tmpl, err := template.New("test").Parse(`
{{.Count}} items are made of {{.Material}}
`)
if err != nil {
panic(err)
}
// 填充模板
err = tmpl.Execute(os.Stdout, sweaters)
if err != nil {
panic(err)
}

输出

1
2
17 items are made of wool

2.if else

1.判断字符串是否相等

1
2
3
4
5
6
7
8
9
10
11
{{ if eq .Material "wool" }}
Material is "wool"
{{ else if eq .Material "not wool" }}
Material is not "wool"
{{ end }}
{{ if eq .Count 17 }}
Count is 17
{{ else if eq .Count 10 }}
Count is not 10
{{ end }}

输出

1
2
3
4
Material is "wool"

Count is 17

2.多条件,与或非

1
2
3
4
{{ if or (eq .Material "not wool") (eq .Count 17) }}
Count is 17
{{ end }}

输出

1
2
Count is 17

3.for循环

1.遍历数组

在for循环中使用其他变量的时候,要用

取得index和value,使用和

1
2
3
4
{{range $index, $value := .Ips}}
index: {{$index}}, material: {{$.Material}}, value: {{$value}}
{{end}}

输出

1
2
3
4
index: 0, material: wool, value: 192.168.0.1

index: 1, material: wool, value: 192.168.0.2

2.遍历map

同样也可以使用for循环遍历map,取得key和value

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
type Inventory struct {
Material string
Count uint
Ips []string
Maps map[string]string
}
sweaters := Inventory{
"wool",
17,
[]string{"192.168.0.1", "192.168.0.2"},
map[string]string{
"key": "hello",
"value": "world",
},
}
tmpl, err := template.New("test").Parse(`
{{range $key, $value := .Maps}}
key: {{$key}}, material: {{$.Material}}, value: {{$value}}
{{end}}
`)
if err != nil {
panic(err)
}
err = tmpl.Execute(os.Stdout, sweaters)
if err != nil {
panic(err)
}

4.函数

1.len函数

判断数组的长度是否等于1,其中Ips可以是切片或者map

1
2
3
4
5
6
7
8
{{if eq (len .Ips) 1}}
len=1
{{else if eq (len .Ips) 2}}
len=2
{{else}}
other
{{end}}

2.index函数

可以使用index函数获得数组特定下标的值

1
2
{{index .Ips 0}}

如果和eq函数一起使用

1
2
3
4
{{if or (eq (index .Ips 0) "192.168.0.1") (eq (index .Ips 1) "192.168.0.2")}}

{{end}}

3.for循环下标从1开始

参考:golang template(数组循环、在循环内使用外部变量、索引从1开始)

4.自定义函数

可以自定义函数来自己实现函数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
package main

import (
"bytes"
"strings"
"text/template"
)

func main() {
type Inventory struct {
Material string
Count uint
Ips []string
Maps map[string]string
}
sweaters := Inventory{
"test1,test2",
17,
[]string{"192.168.0.1", "192.168.0.2"},
map[string]string{
"key": "hello",
"value": "world",
},
}
tmpl := template.New("test")
funcs := template.FuncMap{
"hasSuffix": strings.HasSuffix,
"split": strings.Split,
"containItem": containItem,
"renderTemplate": func(name string, data interface{}) (string, error) {
var buf bytes.Buffer
err := tmpl.ExecuteTemplate(&amp;buf, name, data)
return buf.String() + " test", err
},
}
temp, err := tmpl.Funcs(funcs).Parse(
`
{{if hasSuffix .Material "test2"}}
hasSuffix
{{end}}
`)
if err != nil {
panic(err)
}
var output bytes.Buffer
err = temp.Execute(&amp;output, sweaters)
if err != nil {
panic(err)
}
str := output.String()
println(str)
}

func containItem(slice []string, item string) bool {
for _, v := range slice {
if v == item {
return true
}
}
return false
}

判断字符串是否包含特定后缀

1
2
3
4
{{if hasSuffix .Material "test2"}}
hasSuffix
{{end}}

将字符串以特定字符切分成数组,然后遍历

1
2
3
4
5
{{- $items := split .Material "," -}}
{{- range $index, $item := $items -}}
{{$item}}
{{end}}

输出

1
2
3
test1
test2

判断切片是否包含特定字符串

1
2
3
4
{{if containItem .Ips "192.168.0.1"}}
containItem
{{end}}

5.其他

1.注释

1
2
{{/* 注释 */}}

2.子模板

可以在template中使用define关键字定义一个子模板,然后使用template关键字添加这个子模板

1
2
3
4
5
6
7
start define a template
{{- define "T1" -}}
define a template
{{- end -}}
{{template "T1" .}}
end define a template

输出

1
2
3
4
5
6
7
start define a template


define a template

end define a template

3.去除空格

可以看到上面渲染出来的字符串中有很多的空格,可以使用 - 来去掉多余的空格

比如去掉前面的空格

1
2
{{- "start define a template"}}

去掉后面的空格

1
2
{{"start define a template"}}

去掉模板前后,和模板中的空格

1
2
3
4
5
6
7
{{- "start define a template" -}}
{{define "T1" -}}
define a template
{{- end}}
{{- template "T1" . -}}
{{- "end define a template" -}}

输出

1
2
start define a templatedefine a templateend define a template

4.将子模板定义成一个变量,并使用函数进行处理后输出

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
package main

import (
"bytes"
"text/template"
)

func main() {
type Inventory struct {
Material string
Count uint
Ips []string
Maps map[string]string
}
sweaters := Inventory{
"wool",
17,
[]string{"192.168.0.1", "192.168.0.2"},
map[string]string{
"key": "hello",
"value": "world",
},
}
tmpl := template.New("test")
funcs := template.FuncMap{
"renderTemplate": func(name string, data interface{}) (string, error) {
var buf bytes.Buffer
err := tmpl.ExecuteTemplate(&amp;buf, name, data)
return buf.String() + " test", err
},
}
temp, err := tmpl.Funcs(funcs).Parse(
`
{{define "T1" -}}
define a template
{{- end}}

{{define "T111"}}
{{- $message := renderTemplate "T1" . -}}
{{$message}}
{{end}}

{{template "T111" .}}
`)
if err != nil {
panic(err)
}
var output bytes.Buffer
err = temp.Execute(&amp;output, sweaters)
if err != nil {
panic(err)
}
str := output.String()
println(str)
}

全文 >>

go学习笔记——go-redis

官方文档

https://pkg.go.dev/github.com/go-redis/redis/v8#section-readme

添加依赖

1
2
3
go get github.com/go-redis/redis/v8
go get github.com/go-redis/redis/extra/redisotel/v8

初始化client

1
2
3
4
5
6
7
8
9
10
11
12
client := redis.NewClient(&amp;redis.Options{
Addr: "localhost:6379",
Password: "", // no password set
DB: 0, // use default DB
})

client.AddHook(&amp;redisotel.TracingHook{})
if err := client.Ping(context.Background()).Err(); err != nil {
logger.Error("redis connect failed, err:", zap.Any("err", err))
panic("failed to connect redis")
}

set key

1
2
3
4
5
err = rdb.Set(ctx, "key", 10, time.Hour).Err()
if err != nil {
fmt.Println(err)
}

get key

1
2
3
4
5
6
7
result := client.Get(ctx, "key")
str, err := result.Result()
if err != nil {
fmt.Println(err)
}
fmt.Println(str)

参考:Go语言操作Redis

全文 >>

特征平台简介

特征平台(feature store)的定义:一个用于机器学习的数据管理层,允许共享和发现特征并创建更有效的机器学习管道。

1.特征平台业界实现

1.uber

其最早由uber于2017年提出,uber的feature store名为Michaelangelo,参考:Meet Michelangelo: Uber’s Machine Learning Platform

Michaelangelo主要提供以下6种特性,如下图所示:

  1. Manage data
  2. Train models
  3. Evaluate models
  4. Deploy models
  5. Make predictions
  6. Monitor predictions

2.美团

特征平台所能解决的一些问题:

  • 特征迭代成本高:框架缺乏配置化管理,新特征上线需要同时改动离线侧和在线侧代码,迭代周期较长。
  • 特征复用困难:外卖不同业务线间存在相似场景,使特征的复用成为可能,但框架缺乏对复用能力的很好支撑,导致资源浪费、特征价值无法充分发挥。
  • 平台化能力缺失:框架提供了特征读写的底层开发能力,但缺乏对特征迭代完整周期的平台化追踪和管理能力。

参考:美团外卖特征平台的建设与实践

其他美团的文章:美团配送实时特征平台建设实践

3.字节跳动

特征存储所解决的一些问题:

  1. 存储原始特征:由于在线特征抽取在特征调研上的低效率,我们期望能够存储原始特征;

  2. 离线调研能力:在原始特征的基础上,可以进行离线调研,从而提升特征调研效率;

  3. 支持特征回填:支持特征回填,在调研完成后,可以将历史数据全部刷上调研好的特征;

  4. 降低存储成本:充分利用数据分布的特殊性,降低存储成本,腾出资源来存储原始特征;

  5. 降低训练成本:训练时只读需要的特征,而非全量特征,降低训练成本;

  6. 提升训练速度:训练时尽量降低数据的拷贝和序列化反序列化开销。

参考:字节跳动基于 Iceberg 的海量特征存储实践

2.特征拼接问题

1.离线拼接(离线特征)

以天级的batch任务来实现特征和曝光/点击的join,比如

全文 >>