tonglin0325的个人主页

使用python操作Amazon S3

1.判断s3 object是否存在

1
2
3
4
5
6
7
8
9
10
11
import boto3

s3 = boto3.resource('s3')
bucket = s3.Bucket('my-bucket')
key = 'dootdoot.jpg'
objs = list(bucket.objects.filter(Prefix=key))
if any([w.key == path_s3 for w in objs]):
print("Exists!")
else:
print("Doesn't exist")

2.读取s3 object文件内容

1
2
3
4
5
6
import boto3

s3 = boto3.resource('s3')
object_content = s3.Object('my_bucket_name', object_dir)
file_content = object_content.get()['Body'].read().decode('utf-8')

3.列出s3 object目录

1
2
3
4
5
6
7
8
import boto3

s3 = boto3.resource('s3')
my_bucket = s3.Bucket('my_bucket_name')
dirs = sorted([_.key for _ in my_bucket.objects.filter(Prefix="aaa/bbb/ccc/")], reverse=True)
for dir in dirs:
print(dir)

4.查看emr集群信息,参考:https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/emr.html#EMR.Client.describe_cluster

1
2
3
4
5
6
import boto3

emr_client = boto3.client('emr',region_name='us-west-2')
response = emr_client.describe_cluster(ClusterId="j-xxxxxx")
print(response['Cluster']['MasterPublicDnsName'])