企业AI算力成本直线下降！DeepSeek+亚马逊云科技Spot实例的“神仙组合“有多香？附部署全流程

博思云为

913人浏览 · 2025-02-14 09:31:28

博思云为 · 2025-02-14 09:31:28 发布

亚马逊云科技 Spot实例以其显著的成本优势和灵活性，成为企业运行计算密集型任务的理想选择。DeepSeek作为一款强大的AI助手，能够帮助企业快速部署、交付速度快、应用简单，支持多模型集成和主流大模型快速接入，应用场景丰富。因此，当追求极致性价比的中国AI团队DeepSeek遇上亚马逊云科技Spot实例。企业不仅可以享受DeepSeek带来的高效和便捷，还能大幅降低运行成本，实现资源的最优配置。

而本文内容，将为大家带来全流程部署操作手册，如在部署过程中遇到任何问题，均可VX搜索[博思云为]关注后通过文末二维码实时沟通。

要点总结

1、本文目的

本文目的是在亚马逊云科技的eks 服务上去部署 deepseek 模型，并实现本地的调用。

同时eks 在启动实例类型的时候，需指定 Spot 类型，这样可在原有基础上降低企业成本，但计算性能依然保持高效且顺畅。

2、Spot 实例的特点

低成本：Spot 实例的价格通常比按需实例低 70%-90%，适合预算敏感型任务。
灵活性：适用于可以容忍中断的工作负载，因为 AWS 可能会在需要时收回 Spot 实例。
竞价机制：用户可以设置最高愿意支付的价格，AWS 会根据市场价格分配实例。
中断通知：在实例被收回前，AWS 会提供两分钟的中断通知，便于用户保存状态或进行迁移。

使用场景：

批处理任务：如大规模数据处理、视频转码等。
容器化工作负载：通过 Kubernetes 或 ECS 管理的弹性工作负载。
大规模计算：如机器学习训练、基因分析等。
开发和测试环境：非关键性任务的环境搭建。

一、部署 DeepSeek步骤

如果存在 eks 集群，可以直接跳过以下步骤。直接部署 Karpenter插件和 Nvidia插件

二、部署EKS集群

部署安装 eksctl 工具

文档地址：https://eksctl.io/installation/

# 安装 eksctl,配置环境变量ARCH=amd64PLATFORM=$(uname -s)_$ARCH
# 下载curl -sLO "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_$PLATFORM.tar.gz"
# 解压tar -xzf eksctl_$PLATFORM.tar.gz -C /tmp && rm -rf eksctl_$PLATFORM.tar.gz
# 移动可执行目录sudo mv /tmp/eksctl /usr/bin
# 验证[root@ip-192-168-40-201 ~]# eksctl version0.203.0

三、创建EKS集群

文档地址：

https://docs.aws.amazon.com/zh_cn/eks/latest/userguide/quickstart.html

# 生成集群配置 yamlcat >deepseek.yaml<<EOFapiVersion: eksctl.io/v1alpha5kind: ClusterConfigmetadata:  name: deepseek  region: us-east-1  version: "1.30"vpc:  id: "vpc-051ad30906380db63"  subnets:    public:      us-east-1a: { id: subnet-0a1c2b100ac0bc1f9 }      us-east-1b: { id: subnet-0322fa7a61e37d532 }      us-east-1c: { id: subnet-01c03cc8a7370200a }
iam:  withOIDC: true
addons: - name: vpc-cni - name: coredns - name: kube-proxy
managedNodeGroups:  - name: deepseek-ng    amiFamily: AmazonLinux2    labels: { role: workers }    instanceType: t3a.large    desiredCapacity: 2    minSize: 0    maxSize: 10    volumeSize: 50    volumeType: gp3    maxPodsPerNode: 58    ssh:      allow: true      publicKeyName: raymondEOF

# eks 集群创建eksctl create cluster -f deepseek.yaml

四、安装 alb-ingress 插件

这个步骤主要是用于后续的访问。因为 deepseek 的应用程序部署出来之后，需要提供对外访问

参考文档：

https://docs.aws.amazon.com/zh_cn/eks/latest/userguide/lbc-helm.html

# 准备策略curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/v2.11.0/docs/install/iam_policy.json
# 创建策略aws iam create-policy \    --policy-name AWSLoadBalancerControllerIAMPolicy \    --policy-document file://iam_policy.json
# 创建 sa 与绑定 roleeksctl create iamserviceaccount \  --cluster=my-cluster \  --namespace=kube-system \  --name=aws-load-balancer-controller \  --role-name AmazonEKSLoadBalancerControllerRole \  --attach-policy-arn=arn:aws:iam::111122223333:policy/AWSLoadBalancerControllerIAMPolicy \  --approve
# helm添加repohelm repo add eks https://aws.github.io/eks-chartshelm repo update eks
# 执行部署helm install aws-load-balancer-controller eks/aws-load-balancer-controller \  -n kube-system \  --set clusterName=my-cluster \  --set serviceAccount.create=false \  --set serviceAccount.name=aws-load-balancer-controller

# 验证是否已经安装kubectl get deployment -n kube-system aws-load-balancer-controller

五、集群部署 Karpenter

配置karpenter 的 role

部署 Karpenter 的目的是为了自动扩容 Spot 实例或者 On-demand 实例来运行应用。

官网地址：

https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/

# 配置环境变量KARPENTER_NAMESPACE=kube-systemCLUSTER_NAME=deepseekAWS_PARTITION="aws"AWS_REGION="$(aws configure list | grep region | tr -s " " | cut -d" " -f3)"OIDC_ENDPOINT="$(aws eks describe-cluster --name "${CLUSTER_NAME}" \    --query "cluster.identity.oidc.issuer" --output text)"AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' \    --output text)K8S_VERSION=1.30ARM_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-arm64/recommended/image_id --query Parameter.Value --output text)"AMD_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2/recommended/image_id --query Parameter.Value --output text)"GPU_AMI_ID="$(aws ssm get-parameter --name /aws/service/eks/optimized-ami/${K8S_VERSION}/amazon-linux-2-gpu/recommended/image_id --query Parameter.Value --output text)"

# 验证echo "$KARPENTER_NAMESPACE" "$CLUSTER_NAME"  "$AWS_PARTITION"  "$AWS_REGION"  "$OIDC_ENDPOINT"  "$AWS_ACCOUNT_ID" "$K8S_VERSION" "$ARM_AMI_ID" "$AMD_AMI_ID" "$GPU_AMI_ID"
#运行Cloudformation，自动创建集群所需要的 policy，rolecurl -fsSL https://raw.githubusercontent.com/aws/karpenter-provider-aws/v"${KARPENTER_VERSION}"/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml > "${TEMPOUT}" \&& aws cloudformation deploy \--stack-name "Karpenter-${CLUSTER_NAME}" \--template-file "${TEMPOUT}" \--capabilities CAPABILITY_NAMED_IAM \--parameter-overrides "ClusterName=${CLUSTER_NAME}"#创建控制器的 roleeksctl create iamserviceaccount \--cluster=${CLUSTER_NAME} \--namespace=${KARPENTER_NAMESPACE} \--name=karpenter \--role-name=KarpenterControllerRole-${CLUSTER_NAME} \--attach-policy-arn=arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME} \--approve

六、安全组、子网打标签

# 为现有的nodegroup所在的子网打标签for NODEGROUP in $(aws eks list-nodegroups --cluster-name ${CLUSTER_NAME} \    --query 'nodegroups' --output text); do aws ec2 create-tags \        --tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}" \        --resources $(aws eks describe-nodegroup --cluster-name ${CLUSTER_NAME} \        --nodegroup-name $NODEGROUP --query 'nodegroup.subnets' --output text )done
# 查询nodegroupNODEGROUP=$(aws eks list-nodegroups --cluster-name ${CLUSTER_NAME} \    --query 'nodegroups[0]' --output text)# 查询aws ec2 launch templateLAUNCH_TEMPLATE=$(aws eks describe-nodegroup --cluster-name ${CLUSTER_NAME} \    --nodegroup-name ${NODEGROUP} --query 'nodegroup.launchTemplate.{id:id,version:version}' \    --output text | tr -s "\t" ",")# 查询security groupSECURITY_GROUPS=$(aws eks describe-cluster \    --name ${CLUSTER_NAME} --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output text)
# 为安全组打tagaws ec2 create-tags \    --tags "Key=karpenter.sh/discovery,Value=${CLUSTER_NAME}" \    --resources ${SECURITY_GROUPS}

七、configmap映射

# 更新 configmapeksctl create iamidentitymapping \  --cluster ${CLUSTER_NAME} \  --region ${AWS_REGION} \  --arn arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME} \  --group system:nodes \  --group system:bootstrappers \  --username system:node:{{EC2PrivateDNSName}}

八、安装 Karpenter 应用

export KARPENTER_VERSION="1.2.1"
helm template karpenter oci://public.ecr.aws/karpenter/karpenter --version "${KARPENTER_VERSION}" --namespace "${KARPENTER_NAMESPACE}" \    --set "settings.clusterName=${CLUSTER_NAME}" \    --set "settings.interruptionQueue=${CLUSTER_NAME}" \    --set "serviceAccount.annotations.eks\.amazonaws\.com/role-arn=arn:${AWS_PARTITION}:iam::${AWS_ACCOUNT_ID}:role/KarpenterControllerRole-${CLUSTER_NAME}" \    --set controller.resources.requests.cpu=1 \    --set controller.resources.requests.memory=1Gi \    --set controller.resources.limits.cpu=1 \    --set controller.resources.limits.memory=1Gi > karpenter.yaml

# 修改节点亲和affinity:  nodeAffinity:    requiredDuringSchedulingIgnoredDuringExecution:      nodeSelectorTerms:      - matchExpressions:        - key: karpenter.sh/nodepool          operator: DoesNotExist        - key: eks.amazonaws.com/nodegroup          operator: In          values:          - ${NODEGROUP}  podAntiAffinity:    requiredDuringSchedulingIgnoredDuringExecution:      - topologyKey: "kubernetes.io/hostname"
# 部署CRD资源kubectl create -f \    "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_nodepools.yaml"kubectl create -f \    "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml"kubectl create -f \    "https://raw.githubusercontent.com/aws/karpenter-provider-aws/v${KARPENTER_VERSION}/pkg/apis/crds/karpenter.sh_nodeclaims.yaml"
# 部署 karpenterkubectl apply -f karpenter.yaml
# 查看Karpenter的信息[root@ip-192-168-40-201 ~]# k get pods -n kube-system |grep karpenterkarpenter-5794c6fc46-lpbd6                      1/1     Running   0          14hkarpenter-5794c6fc46-sc8wt                      1/1     Running   0          20h

九、建立节点池-默认

# 创建一个默认的节点池，给一般应用使用cat >>default-ec2nc.yaml<<EFOapiVersion: karpenter.sh/v1kind: NodePoolmetadata:  name: defaultspec:  template:    spec:      requirements:        - key: kubernetes.io/arch          operator: In          values: ["amd64"]        - key: kubernetes.io/os          operator: In          values: ["linux"]        - key: karpenter.sh/capacity-type          operator: In          values: ["spot"]        - key: karpenter.k8s.aws/instance-category          operator: In          values: ["c", "m", "r"]        - key: karpenter.k8s.aws/instance-generation          operator: Gt          values: ["2"]      nodeClassRef:        group: karpenter.k8s.aws        kind: EC2NodeClass        name: default      expireAfter: 720h # 30 * 24h = 720h  limits:    cpu: 1000  disruption:    consolidationPolicy: WhenEmptyOrUnderutilized    consolidateAfter: 1m---apiVersion: karpenter.k8s.aws/v1kind: EC2NodeClassmetadata:  name: defaultspec:  amiFamily: AL2 # Amazon Linux 2  role: "KarpenterNodeRole-deepseek" # replace with your cluster name  subnetSelectorTerms:    - tags:        karpenter.sh/discovery: "deepseek" # replace with your cluster name  securityGroupSelectorTerms:    - tags:        karpenter.sh/discovery: "deepseek" # replace with your cluster name  amiSelectorTerms:    - id: "ami-07f0a903b02947a1c"    - id: "ami-0e4591ba595196441"EOF# 创建节点池kubectl apply -f nodepool.yaml

十、建立节点池-gpu（Spot 类型）

cat >>gpu-nodepool.yaml<<EOF apiVersion: karpenter.sh/v1kind: NodePoolmetadata:  name: deepseek-gpuspec:  template:    spec:      requirements:        - key: kubernetes.io/arch          operator: In          values: ["amd64"]        - key: kubernetes.io/os          operator: In          values: ["linux"]        - key: karpenter.sh/capacity-type          operator: In          values: ["spot"]        - key: karpenter.k8s.aws/instance-generation          operator: Gt          values: ["2"]
        - key: "karpenter.k8s.aws/instance-family"          operator: In          values: ["g5","g6","g6e"]      taints:        - key: nvidia.com/gpu          effect: NoSchedule      nodeClassRef:        group: karpenter.k8s.aws        kind: EC2NodeClass        name: deepseek-gpu      expireAfter: 720h # 30 * 24h = 720h  limits:    cpu: 1000  disruption:    consolidationPolicy: WhenEmptyOrUnderutilized    consolidateAfter: 1m---apiVersion: karpenter.k8s.aws/v1kind: EC2NodeClassmetadata:  name: deepseek-gpuspec:  amiFamily: AL2 # Amazon Linux 2  role: "KarpenterNodeRole-deepseek" # replace with your cluster name  subnetSelectorTerms:    - tags:        karpenter.sh/discovery: "deepseek" # replace with your cluster name  securityGroupSelectorTerms:    - tags:        karpenter.sh/discovery: "deepseek" # replace with your cluster name  amiSelectorTerms:    - id: "ami-0dd4d67779d66436c" # <- GPU Optimized AMD AMI  blockDeviceMappings:    - deviceName: /dev/xvda      ebs:        volumeSize: 200Gi        volumeType: gp3        encrypted: trueEOF
# 创建 gpu节点池kubectl apply -f gpu-nodepool.yaml
# 查看节点池[root@ip-192-168-40-201 deepseek]# k get nodepoolNAME           NODECLASS      NODES   READY   AGEdeepseek-gpu   deepseek-gpu   1       True    19hdefault        default        0       True    21h[root@ip-192-168-40-201 deepseek]# [root@ip-192-168-40-201 deepseek]#

十一、部署 nvidia插件

参考文档：https://github.com/NVIDIA/k8s-device-plugin

kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.0/deployments/static/nvidia-device-plugin.yml

十二、部署 DeepSeek 程序

cat >>deepseek.yaml<<EOFapiVersion: v1kind: Namespacemetadata:  name: deepseek---apiVersion: apps/v1kind: Deploymentmetadata:  name: deepseek-deployment  namespace: deepseek  labels:    app: deepseekspec:  replicas: 1  selector:    matchLabels:      app: deepseek  template:    metadata:      labels:        app: deepseek    spec:      tolerations:        - key: "nvidia.com/gpu"          operator: "Exists"          effect: "NoSchedule"        - key: "deepseek-exclusive"          operator: "Equal"          value: "true"          effect: "NoSchedule"      volumes:        - name: cache-volume          hostPath:            path: /tmp/deepseek            type: DirectoryOrCreate        - name: shm          emptyDir:            medium: Memory            sizeLimit: "4Gi"      containers:        - name: deepseek          image: vllm/vllm-openai:latest          command: ["/bin/sh", "-c"]          args: [            "vllm serve deepseek-ai/DeepSeek-R1-Distill-Llama-8B --max_model_len 2048 --tensor-parallel-size 4 --port 8000"          ]          env:            - name: HF_HUB_ENABLE_HF_TRANSFER              value: "0"            - name: disable_custom_all_reduce              value: "True"  # 添加此行以禁用自定义 allreduce            - name: OMP_NUM_THREADS              value: "1"          ports:            - containerPort: 8000          resources:            requests:              nvidia.com/gpu: "4"            limits:              nvidia.com/gpu: "4"          volumeMounts:            - mountPath: /root/.cache/huggingface              name: cache-volume            - name: shm              mountPath: /dev/shm
EOF

十三、服务暴露

cat >>deepseek-svc.yaml<<EOF---apiVersion: v1kind: Servicemetadata:  name: deepseek-svc  namespace: deepseek  annotations:    service.beta.kubernetes.io/aws-load-balancer-type: "external"    service.beta.kubernetes.io/aws-load-balancer-scheme: "internet-facing"    service.beta.kubernetes.io/aws-load-balancer-subnets: "subnet-0a1c2b100ac0bc1f9,subnet-0322fa7a61e37d532,subnet-01c03cc8a7370200a"     service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"spec:  ports:    - port: 80      targetPort: 8000      protocol: TCP  type: LoadBalancer  selector:    app: deepseek
EOF

十四、查看应用启动日志

...INFO 02-12 13:03:18 worker.py:266] Memory profiling takes 2.80 secondsINFO 02-12 13:03:18 worker.py:266] the current vLLM instance can use total_gpu_memory (22.18GiB) x gpu_memory_utilization (0.90) = 19.97GiBINFO 02-12 13:03:18 worker.py:266] model weights take 13.77GiB; non_torch_memory takes 0.21GiB; PyTorch activation peak memory takes 1.19GiB; the rest of the memory reserved for KV Cache is 14.80GiB.INFO 02-12 13:03:18 executor_base.py:108] # CUDA blocks: 30314, # CPU blocks: 8192INFO 02-12 13:03:18 executor_base.py:113] Maximum concurrency for 2048 tokens per request: 236.83x(VllmWorkerProcess pid=39) INFO 02-12 13:03:21 model_runner.py:1435] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.(VllmWorkerProcess pid=40) INFO 02-12 13:03:21 model_runner.py:1435] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.INFO 02-12 13:03:21 model_runner.py:1435] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.(VllmWorkerProcess pid=38) INFO 02-12 13:03:21 model_runner.py:1435] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.(VllmWorkerProcess pid=38) INFO 02-12 13:03:39 model_runner.py:1563] Graph capturing finished in 18 secs, took 0.43 GiBCapturing CUDA graph shapes: 100%|██████████| 35/35 [00:17<00:00,  1.97it/s]INFO 02-12 13:03:39 model_runner.py:1563] Graph capturing finished in 18 secs, took 0.43 GiB(VllmWorkerProcess pid=40) INFO 02-12 13:03:39 model_runner.py:1563] Graph capturing finished in 18 secs, took 0.43 GiB(VllmWorkerProcess pid=39) INFO 02-12 13:03:39 model_runner.py:1563] Graph capturing finished in 18 secs, took 0.43 GiBINFO 02-12 13:03:39 llm_engine.py:429] init engine (profile, create kv cache, warmup model) took 23.99 secondsINFO 02-12 13:03:40 api_server.py:754] Using supplied chat template:INFO 02-12 13:03:40 api_server.py:754] NoneINFO 02-12 13:03:40 launcher.py:19] Available routes are:INFO 02-12 13:03:40 launcher.py:27] Route: /openapi.json, Methods: HEAD, GETINFO 02-12 13:03:40 launcher.py:27] Route: /docs, Methods: HEAD, GETINFO 02-12 13:03:40 launcher.py:27] Route: /docs/oauth2-redirect, Methods: HEAD, GETINFO 02-12 13:03:40 launcher.py:27] Route: /redoc, Methods: HEAD, GETINFO 02-12 13:03:40 launcher.py:27] Route: /health, Methods: GETINFO 02-12 13:03:40 launcher.py:27] Route: /ping, Methods: POST, GETINFO 02-12 13:03:40 launcher.py:27] Route: /tokenize, Methods: POSTINFO 02-12 13:03:40 launcher.py:27] Route: /detokenize, Methods: POSTINFO 02-12 13:03:40 launcher.py:27] Route: /v1/models, Methods: GETINFO 02-12 13:03:40 launcher.py:27] Route: /version, Methods: GETINFO 02-12 13:03:40 launcher.py:27] Route: /v1/chat/completions, Methods: POSTINFO 02-12 13:03:40 launcher.py:27] Route: /v1/completions, Methods: POSTINFO 02-12 13:03:40 launcher.py:27] Route: /v1/embeddings, Methods: POSTINFO 02-12 13:03:40 launcher.py:27] Route: /pooling, Methods: POSTINFO 02-12 13:03:40 launcher.py:27] Route: /score, Methods: POSTINFO 02-12 13:03:40 launcher.py:27] Route: /v1/score, Methods: POSTINFO 02-12 13:03:40 launcher.py:27] Route: /rerank, Methods: POSTINFO 02-12 13:03:40 launcher.py:27] Route: /v1/rerank, Methods: POSTINFO 02-12 13:03:40 launcher.py:27] Route: /v2/rerank, Methods: POSTINFO 02-12 13:03:40 launcher.py:27] Route: /invocations, Methods: POSTINFO:     Started server process [7]INFO:     Waiting for application startup.INFO:     Application startup complete.INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)...

十五、功能测试

# 先在本地测试，验证功能curl -X POST \  "http://k8s-deepseek-deepseek-77e813e857-ed7fb94a9df4db5d.elb.us-east-1.amazonaws.com/v1/chat/completions" \  -H "Content-Type: application/json" \  --data '{    "model": "deepseek-ai/DeepSeek-R1-Distill-Llama-32B",    "messages": [      {        "role": "user",        "content": "你是谁"      }   ]  }'
# 内容输出{"id":"chatcmpl-2dd3f3268347495ea99a976341ad522e","object":"chat.completion","created":1739432033,"model":"deepseek-ai/DeepSeek-R1-Distill-Llama-32B","choices":[{"index":0,"message":{"role":"assistant","reasoning_content":null,"content":"你好！我是DeepSeek-R1，一个由深度求索公司开发的智能助手，我会尽我所能为您提供帮助。\n</think>\n\n你好！我是DeepSeek-R1，一个由深度求索公司开发的智能助手，我会尽我所能为您提供帮助。","tool_calls":[]},"logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":8,"total_tokens":77,"completion_tokens":69,"prompt_tokens_details":null},"prompt_logprobs":null}