[Document] 原神启动器的 Chunk 下载模式分析 #725

Scighost · 2024-03-18T14:37:52Z

一直以来，原神官服启动器的下载模式饱受玩家诟病，其采用的压缩包-解压的模式会占用两倍于压缩包的存储空间。从原神 4.5 版本的预下载开始，启动器加入了全新的 Chunk 下载模式，解决了这个缺点。在此模式下，单个游戏资源文件被切分为多个块分别被下载，最后在本地组合成一个整体。

灰度测试

Chunk 下载模式正处于灰度测试中，但是可以通过修改 API 的返回值强制开启。启动器会请求下列 API 若干次：

https://abtest-api-data.mihoyo.com/data_abtest_api/config/experiment/list

在某一次的请求中会有如下的返回值，只需把 "downloadMode": "file" 修改为 "downloadMode": "chunk" 即可启用 Chunk 下载模式。

{
  "retcode": 0,
  "success": true,
  "message": "",
  "data": [
    {
      "code": 1010,
      "type": 2,
      "config_id": "347",
      "period_id": "",
      "version": "",
      "configs": {
        "downloadMode": "file"
      },
      "sceneWhiteList": false,
      "experimentWhiteList": false
    }
  ]
}

版本信息

启动器通过下列 API 获得游戏的版本数据，此外还可以通过添加查询参数 tag=4.5.0 获取特定版本的信息。非预下载期间，预下载 API 的返回值为空。

官服正式版
https://api-takumi.mihoyo.com/downloader/sophon_chunk/api/getBuild?branch=main&package_id=s8m3Yf3j6G&password=QlYzM79uF6va&plat_app=cxgf44wie1a8

官服预下载
https://api-takumi.mihoyo.com/downloader/sophon_chunk/api/getBuild?branch=predownload&package_id=s8m3Yf3j6G&password=izObT6iTHAqq&plat_app=cxgf44wie1a8

B服正式版
https://api-takumi.mihoyo.com/downloader/sophon_chunk/api/getBuild?branch=main&package_id=yxzhR2pRqE&password=mPNvWfvLBO8b&plat_app=cxgf44wie1a8

B服预下载
https://api-takumi.mihoyo.com/downloader/sophon_chunk/api/getBuild?branch=predownload&package_id=yxzhR2pRqE&password=16dFsdfbRBPO&plat_app=cxgf44wie1a8

国际服正式版
https://sg-public-api.hoyoverse.com/downloader/sophon_chunk/api/getBuild?branch=main&package_id=Ul0pnnZo8h&password=SSaKNEvZjrK0&plat_app=cxhpq4g4rgg0

国际服预下载
https://sg-public-api.hoyoverse.com/downloader/sophon_chunk/api/getBuild?branch=predownload&package_id=Ul0pnnZo8h&password=yran8QsvU88T&plat_app=cxhpq4g4rgg0

这里展示官服 4.5.0 版本的信息，JSON 的结构和命名非常清晰，通过节点 manifest 和 manifest_download 可以组合出资源清单文件的下载链接，通过节点 chunk_download 和清单文件中的内容可以组合出游戏资源文件的下载链接。

API 中存在 encryption 和 password 参数，后续可能会对资源文件加密。

版本信息示例

{
    "retcode": 0,
    "message": "OK",
    "data": {
        "build_id": "D7g0SFeiFg9o",
        "tag": "4.5.0",
        "manifests": [
            {
                "category_id": "10017",
                "category_name": "游戏资源-外网",
                "manifest": {
                    "id": "manifest_233f3acd5276c84e_890ab337d4ec8edf6c98c4dcf702b8bf",
                    "checksum": "890ab337d4ec8edf6c98c4dcf702b8bf",
                    "compressed_size": "4736513",
                    "uncompressed_size": "9818860"
                },
                "chunk_download": {
                    "encryption": 0,
                    "password": "",
                    "compression": 1,
                    "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/chunks/cxgf44wie1a8/62nfHL6ocpNF",
                    "url_suffix": ""
                },
                "manifest_download": {
                    "encryption": 0,
                    "password": "",
                    "compression": 1,
                    "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/manifests/cxgf44wie1a8/62nfHL6ocpNF",
                    "url_suffix": ""
                },
                "matching_field": "game",
                "stats": {
                    "compressed_size": "75506162967",
                    "uncompressed_size": "76752839912",
                    "file_count": "16115",
                    "chunk_count": "74156"
                },
                "deduplicated_stats": {
                    "compressed_size": "75443433225",
                    "uncompressed_size": "76650339773",
                    "file_count": "16115",
                    "chunk_count": "74077"
                }
            },
            {
                "category_id": "10018",
                "category_name": "语音包-中文-外网",
                "manifest": {
                    "id": "manifest_fe557261015fe099_f13178635a13eb92613660a2989a8d48",
                    "checksum": "f13178635a13eb92613660a2989a8d48",
                    "compressed_size": "697187",
                    "uncompressed_size": "1333393"
                },
                "chunk_download": {
                    "encryption": 0,
                    "password": "",
                    "compression": 1,
                    "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/chunks/cxgf44wie1a8/62nfHL6ocpNF",
                    "url_suffix": ""
                },
                "manifest_download": {
                    "encryption": 0,
                    "password": "",
                    "compression": 1,
                    "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/manifests/cxgf44wie1a8/62nfHL6ocpNF",
                    "url_suffix": ""
                },
                "matching_field": "zh-cn",
                "stats": {
                    "compressed_size": "11858894718",
                    "uncompressed_size": "13671657756",
                    "file_count": "129",
                    "chunk_count": "11952"
                },
                "deduplicated_stats": {
                    "compressed_size": "11858620147",
                    "uncompressed_size": "13671337332",
                    "file_count": "129",
                    "chunk_count": "11951"
                }
            },
            {
                "category_id": "10019",
                "category_name": "语音包-英文-外网",
                "manifest": {
                    "id": "manifest_3de8d7c6c1a4bef4_1daceaa6e906c0aa3d6621e6039f8b84",
                    "checksum": "1daceaa6e906c0aa3d6621e6039f8b84",
                    "compressed_size": "789952",
                    "uncompressed_size": "1505333"
                },
                "chunk_download": {
                    "encryption": 0,
                    "password": "",
                    "compression": 1,
                    "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/chunks/cxgf44wie1a8/62nfHL6ocpNF",
                    "url_suffix": ""
                },
                "manifest_download": {
                    "encryption": 0,
                    "password": "",
                    "compression": 1,
                    "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/manifests/cxgf44wie1a8/62nfHL6ocpNF",
                    "url_suffix": ""
                },
                "matching_field": "en-us",
                "stats": {
                    "compressed_size": "14728100079",
                    "uncompressed_size": "15959256895",
                    "file_count": "129",
                    "chunk_count": "13500"
                },
                "deduplicated_stats": {
                    "compressed_size": "14727266762",
                    "uncompressed_size": "15958274971",
                    "file_count": "129",
                    "chunk_count": "13499"
                }
            },
            {
                "category_id": "10021",
                "category_name": "语音包-日文-外网",
                "manifest": {
                    "id": "manifest_06dfea23fc844733_2470cf8019c95daa74fe918640414b07",
                    "checksum": "2470cf8019c95daa74fe918640414b07",
                    "compressed_size": "870736",
                    "uncompressed_size": "1666416"
                },
                "chunk_download": {
                    "encryption": 0,
                    "password": "",
                    "compression": 1,
                    "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/chunks/cxgf44wie1a8/62nfHL6ocpNF",
                    "url_suffix": ""
                },
                "manifest_download": {
                    "encryption": 0,
                    "password": "",
                    "compression": 1,
                    "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/manifests/cxgf44wie1a8/62nfHL6ocpNF",
                    "url_suffix": ""
                },
                "matching_field": "ja-jp",
                "stats": {
                    "compressed_size": "14072069356",
                    "uncompressed_size": "17912647294",
                    "file_count": "129",
                    "chunk_count": "14959"
                },
                "deduplicated_stats": {
                    "compressed_size": "14072069356",
                    "uncompressed_size": "17912647294",
                    "file_count": "129",
                    "chunk_count": "14959"
                }
            },
            {
                "category_id": "10020",
                "category_name": "语音包-韩文-外网",
                "manifest": {
                    "id": "manifest_3da522d853512954_f3b98736693bf599efb3d14cdf8717cc",
                    "checksum": "f3b98736693bf599efb3d14cdf8717cc",
                    "compressed_size": "657108",
                    "uncompressed_size": "1255144"
                },
                "chunk_download": {
                    "encryption": 0,
                    "password": "",
                    "compression": 1,
                    "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/chunks/cxgf44wie1a8/62nfHL6ocpNF",
                    "url_suffix": ""
                },
                "manifest_download": {
                    "encryption": 0,
                    "password": "",
                    "compression": 1,
                    "url_prefix": "https://autopatchcn.yuanshen.com/client_app/sophon/manifests/cxgf44wie1a8/62nfHL6ocpNF",
                    "url_suffix": ""
                },
                "matching_field": "ko-kr",
                "stats": {
                    "compressed_size": "11594504922",
                    "uncompressed_size": "13434855981",
                    "file_count": "129",
                    "chunk_count": "11244"
                },
                "deduplicated_stats": {
                    "compressed_size": "11594504922",
                    "uncompressed_size": "13434855981",
                    "file_count": "129",
                    "chunk_count": "11244"
                }
            }
        ]
    }
}

在此 API 中，官服和B服的内容完全一致，B服的 PCGameSDK.dll 仍需通过以下 API 获取：

https://hk4e-launcher-static.mihoyo.com/hk4e_cn/mdk/launcher/api/resource?channel_id=14&key=KAtdSsoQ&launcher_id=17&sub_channel_id=0

资源清单

游戏资源清单和游戏资源文件均使用 Zstandard (zstd) 压缩算法，清单文件在解压后需要使用 Protocol Buffers (protobuf) 解析其内容。

这里使用 .NET protobuf-net 的形式展示其数据模型

List<GameFile> manifest = Serializer.Deserialize<List<GameFile>>(manifest_bytes);

/// <summary>
/// 游戏资源文件
/// </summary>
[ProtoContract]
public class GameFile
{
    /// <summary>
    /// 文件或文件夹相对于游戏根目录的路径
    /// </summary>
    [ProtoMember(1)]
    public string File { get; set; }

    [ProtoMember(2)]
    public List<Chunk> Chunks { get; set; }

    [ProtoMember(3)]
    public bool IsFolder { get; set; }

    [ProtoMember(4)]
    public long Size { get; set; }

    [ProtoMember(5)]
    public string MD5 { get; set; }
}

/// <summary>
/// 资源文件块
/// </summary>
[ProtoContract]
public class Chunk
{
    /// <summary>
    /// 文件块的下载链接后缀
    /// </summary>
    [ProtoMember(1)]
    public string UrlSuffix { get; set; }

    /// <summary>
    /// 文件块解压后的 MD5 值
    /// </summary>
    [ProtoMember(2)]
    public string MD5 { get; set; }

    /// <summary>
    /// 文件块解压后，相对于文件头的偏移量
    /// </summary>
    [ProtoMember(3)]
    public long Offset { get; set; }

    /// <summary>
    /// 解压前的大小
    /// </summary>
    [ProtoMember(4)]
    public long CompressedSize { get; set; }

    /// <summary>
    /// 解压后的块大小
    /// </summary>
    [ProtoMember(5)]
    public long Size { get; set; }

    /// <summary>
    /// 未知哈希算法
    /// </summary>
    [ProtoMember(6)]
    public ulong Unknown { get; set; }
}

资源文件示例

{
    "File": "YuanShen_Data/StreamingAssets/AssetBundles/blocks/00/05130550.blk",
    "Chunks": [
        {
            "Suffix": "fc9327bfb71c7a8a_5a8d03f4c687b6808304166e9babde95",
            "MD5": "5a8d03f4c687b6808304166e9babde95",
            "Offset": 0,
            "CompressedSize": 1754811,
            "Size": 1754756,
            "Unknown": 15817881018744700928
        },
        {
            "Suffix": "88ea771167162221_ff7e21a86b098f3ee055c08e1d2b44a4",
            "MD5": "ff7e21a86b098f3ee055c08e1d2b44a4",
            "Offset": 1754756,
            "CompressedSize": 1208916,
            "Size": 1208873,
            "Unknown": 1333711227053146112
        },
        {
            "Suffix": "b9b9aaf28252291b_8c01b876372f43bd50694236b352bd3c",
            "MD5": "8c01b876372f43bd50694236b352bd3c",
            "Offset": 2963629,
            "CompressedSize": 83939,
            "Size": 83923,
            "Unknown": 0
        }
    ],
    "IsFolder": false,
    "Size": 3047552,
    "MD5": "e89988407ded973318bf8298976bfcf8"
}

下载步骤

首先下载游戏资源文件块到根目录的 chunk 文件夹中，此时文件块还未被解压，文件夹中也不存在子文件夹。但是，下载后的文件块名称并不和 MD5 值相同，命名方式我没有分析出来。

接下来解压对应的文件块，在 staging 文件夹中组合成一个完整的游戏资源文件，此文件夹中存在着和游戏根目录相同的子文件夹结构。组合完成后再移动资源文件到正确的路径。

而在预下载过程中，对比新旧两个版本的资源清单，仅下载有差异的部分。

总结

相比于老启动器的 File 下载模式，新的 Chunk 下载模式解决了需要预留两倍存储空间的问题，下载过程中也能更好地利用多线程提高下载速度。但是版本更新过程中音频资源文件不再使用 HDiffPatch，下载的数据量相比之前有所增加。

The text was updated successfully, but these errors were encountered:

bangbang23333 · 2024-04-18T15:48:47Z

encryption 和 password 参数应该是给测试服包体下载用的，测试服启动器是要登录账号的

prpjzz · 2024-04-23T11:20:47Z

I have two questions like this:

Can we apply this to Starward?
What's the old game compression algorithm for blk files?

Lightczx · 2024-04-24T01:00:20Z

我来构成交叉引用 https://blog.amarea.cn/archives/genshin-launcher-chunk-download-mode.html

Lightczx · 2024-04-24T01:09:51Z

chunk.db

-- auto-generated definition
create table depot_file_data
(
    file_main_key LONGVARCHAR
        primary key,
    package_id    LONGVARCHAR,
    build_id      LONGVARCHAR,
    depot_id      LONGVARCHAR,
    file_id       LONGVARCHAR,
    file_ver      LONGVARCHAR,
    file_status   INTEGER default 0,
    install_dir   LONGVARCHAR
);

chunk_config.db

-- auto-generated definition
create table config
(
    package_id_branch_depot_id LONGVARCHAR
        primary key,
    local_version              LONGVARCHAR,
    server_version             LONGVARCHAR,
    local_build_id             LONGVARCHAR,
    server_build_id            LONGVARCHAR,
    package_id                 LONGVARCHAR,
    branch                     LONGVARCHAR,
    depot_id                   LONGVARCHAR,
    matching_field             LONGVARCHAR,
    install_dir                LONGVARCHAR
);

chunk_manifest.db

-- auto-generated definition
create table depot_manifest
(
    package_id_build_id_depot_id     LONGVARCHAR
        primary key,
    package_id                       LONGVARCHAR,
    build_id                         LONGVARCHAR,
    depot_id                         LONGVARCHAR,
    version                          LONGVARCHAR,
    depot_manifest_id                LONGVARCHAR,
    depot_manifest_checksum          LONGVARCHAR,
    depot_manifest_compressed_size   INTEGER default 0,
    depot_manifest_uncompressed_size INTEGER default 0,
    depot_manifest_url_prefix        LONGVARCHAR,
    depot_manifest_encryption        INTEGER default 0,
    depot_manifest_password          LONGVARCHAR,
    depot_manifest_compression       INTEGER default 0,
    depot_manifest_md5               LONGVARCHAR
);

Scighost added documentation Improvements or additions to documentation enhancement New feature or request Game: Genshin Area: Install Game labels Mar 18, 2024

Scighost mentioned this issue Mar 18, 2024

Changes to the way download works #698

Closed

2 tasks

Scighost mentioned this issue Jun 3, 2024

[Feature] Add Sophon Dowloader #811

Closed

2 tasks

HongchengQ mentioned this issue Jun 18, 2024

启动器的 Chunk 下载模式分析 HongchengQ/Hyp-ChunkDownload-Library#1

Closed

Eric-Joker mentioned this issue Jul 2, 2024

[Feature] 包体流式下载解压 #861

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Document] 原神启动器的 Chunk 下载模式分析 #725

[Document] 原神启动器的 Chunk 下载模式分析 #725

Scighost commented Mar 18, 2024

bangbang23333 commented Apr 18, 2024

prpjzz commented Apr 23, 2024

Lightczx commented Apr 24, 2024

Lightczx commented Apr 24, 2024 •

edited

Loading

[Document] 原神启动器的 Chunk 下载模式分析 #725

[Document] 原神启动器的 Chunk 下载模式分析 #725

Comments

Scighost commented Mar 18, 2024

灰度测试

版本信息

资源清单

下载步骤

总结

bangbang23333 commented Apr 18, 2024

prpjzz commented Apr 23, 2024

Lightczx commented Apr 24, 2024

Lightczx commented Apr 24, 2024 • edited Loading

Lightczx commented Apr 24, 2024 •

edited

Loading