feat(core): Upgrade pydantic to 2.x

[fangyinc]

Description

Closes #1411

The main upgrades are as follows

• "pydantic<2,>=1" to "pydantic>=2.6.0"
• "fastapi==0.98.0" to "fastapi>=0.100.0"
• "spacy==3.5.3" to "spacy>=3.7"
• "chromadb==0.4.10" to "chromadb>=0.4.22"

Note:
Please create a new conda environment, and then use the commandpip install -e ".[default]" to reinstall the dependencies.

How Has This Been Tested?

We need comprehensive regression testing!!!

Snapshots:

Include snapshots for easier review.

Checklist:

• My code follows the style guidelines of this project
• I have already rebased the commits and make the commit message conform to the project standard.
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• Any dependent changes have been merged and published in downstream modules

[csunny]

When i test this feature use MacOS, some error occur

Traceback (most recent call last):
  File "/Users/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/app/dbgpt_server.py", line 242, in <module>
    run_webserver()
  File "/Users/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/app/dbgpt_server.py", line 237, in run_webserver
    param = initialize_app(param)
  File "/Users/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/app/dbgpt_server.py", line 152, in initialize_app
    initialize_components(param, system_app, embedding_model_name, embedding_model_path)
  File "/Users/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/app/component_configs.py", line 45, in initialize_components
    _initialize_embedding_model(
  File "/Users/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/app/initialization/embedding_component.py", line 28, in _initialize_embedding_model
    system_app.register(
  File "/Users/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/component.py", line 191, in register
    instance = component(self, *args, **kwargs)
  File "/Users/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/app/initialization/embedding_component.py", line 72, in __init__
    self._model = self._load_model()
  File "/Users/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/app/initialization/embedding_component.py", line 107, in _load_model
    return loader.load(self._default_model_name, model_params)
  File "/Users/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/model/adapter/embeddings_loader.py", line 57, in load
    return HuggingFaceEmbeddings(**kwargs)
  File "/Users/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/rag/embedding/embeddings.py", line 84, in __init__
    super().__init__(**kwargs)
  File "/Users/magic/miniconda3/envs/dbgpt_env/lib/python3.10/site-packages/pydantic/main.py", line 175, in __init__
    self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for HuggingFaceEmbeddings
client
  Field required [type=missing, input_value={'model_name': '/Users/ma...rgs': {'device': 'mps'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.7/v/missing

The same error about CPU

Traceback (most recent call last):
  File "/home/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/app/dbgpt_server.py", line 242, in <module>
    run_webserver()
  File "/home/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/app/dbgpt_server.py", line 237, in run_webserver
    param = initialize_app(param)
  File "/home/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/app/dbgpt_server.py", line 152, in initialize_app
    initialize_components(param, system_app, embedding_model_name, embedding_model_path)
  File "/home/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/app/component_configs.py", line 45, in initialize_components
    _initialize_embedding_model(
  File "/home/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/app/initialization/embedding_component.py", line 28, in _initialize_embedding_model
    system_app.register(
  File "/home/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/component.py", line 191, in register
    instance = component(self, *args, **kwargs)
  File "/home/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/app/initialization/embedding_component.py", line 72, in __init__
    self._model = self._load_model()
  File "/home/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/app/initialization/embedding_component.py", line 107, in _load_model
    return loader.load(self._default_model_name, model_params)
  File "/home/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/model/adapter/embeddings_loader.py", line 57, in load
    return HuggingFaceEmbeddings(**kwargs)
  File "/home/magic/workspace/github/eosphoros-ai/DB-GPT/dbgpt/rag/embedding/embeddings.py", line 84, in __init__
    super().__init__(**kwargs)
  File "/home/magic/miniconda3/envs/dbgpt_env/lib/python3.10/site-packages/pydantic/main.py", line 175, in __init__
    self.__pydantic_validator__.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for HuggingFaceEmbeddings
client
  Field required [type=missing, input_value={'model_name': '/home/mag...rgs': {'device': 'cpu'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.7/v/missing

[csunny]

Also cuda

pydantic_core._pydantic_core.ValidationError: 1 validation error for HuggingFaceEmbeddings
client
  Field required [type=missing, input_value={'model_name': '/home/mag...gs': {'device': 'cuda'}}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.7/v/missing

[fangyinc]

@csunny @Aries-ckt fixed.

[Aries-ckt]

Test success!

[Aries-ckt]

r+

[Aralhi]

r+