Skip to content

Bump oneapi-basekit, optimum and openvino#2139

Merged
mudler merged 2 commits into
mudler:masterfrom
fakezeta:master
Apr 26, 2024
Merged

Bump oneapi-basekit, optimum and openvino#2139
mudler merged 2 commits into
mudler:masterfrom
fakezeta:master

Conversation

@fakezeta

Copy link
Copy Markdown
Contributor

Description

Bump to the latest release: openvino 2024.1.0

Selected from release notes:

  • Mixtral and URLNet models optimized for performance improvements on Intel® Xeon® processors.
  • The preview NPU plugin for Intel® Core™ Ultra processors is now available in the OpenVINO open-source GitHub repository, in addition to the main OpenVINO package on PyPI.
  • Significant memory reduction for select smaller GenAI models on Intel® Core™ Ultra processors with integrated GPU.
  • (CPU) LLM compilation time and memory footprint have been improved through additional optimizations with compressed embeddings.
  • (CPU) Performance of MoE (e.g. Mixtral), Gemma, and GPT-J has been improved further.
  • (GPU) LLM first token latency has been improved on both integrated and discrete GPU platforms.
  • NPU Device Plugin

Planning to enable NPU selection on transformer backend soon©️

Note for reviewers
It's my first PR bumping all this dependencies please review carefully 😄

  • Yes, I signed my commits.

@netlify

netlify Bot commented Apr 25, 2024

Copy link
Copy Markdown

Deploy Preview for localai canceled.

Name Link
🔨 Latest commit 2a2b15b
🔍 Latest deploy log https://app.netlify.com/sites/localai/deploys/662ae65bebc1610008336bc3

Minor latency change for first token but about 10-15% speedup on token generation.
@fakezeta

Copy link
Copy Markdown
Contributor Author

I've added also a change to PERFORMANCE HINT to CUMULATIVE THROUGHPUT improving inference speed noticeably.

platforms: 'linux/amd64'
tag-latest: 'false'
base-image: "intel/oneapi-basekit:2024.0.1-devel-ubuntu22.04"
base-image: "intel/oneapi-basekit:2024.1.0-devel-ubuntu22.04"

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll have to give a test at this, but I don't see anything wrong in bumping the image

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very curious to know how it works on Arc.
Still some issue to sort with Llama3 while Phi3-128k is working like a charm.

@mudler
mudler merged commit c9451cb into mudler:master Apr 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants