Skip to content

Releases

b12500

b12500 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 18 Nov 07:04
aaa807c

cuda: decode the conv2d patch indices with fastdiv (#32492)

The implicit GEMM conv2d kernel recovers the channel and the kernel
position of each reduction index with three integer divisions by
runtime divisors, once per loaded row of every tile. Outside the 3x3
and 1x1 specializations this integer division bounds the kernel:
removing it halves the time on long reductions.

Divide with the precomputed magic numbers of fastdiv instead. The 3x3
and 1x1 specializations keep their compile time divisors.

test-backend-ops -o CONV_2D: 2006/2006 on CUDA0.

CONV_2D, F16 kernel, us/run on an RTX PRO 6000:
[2054,1,768] k [7,1,768,768] 221 -> 138
[16390,1,384] k [7,1,384,384] 433 -> 266
[65542,1,192] k [7,1,192,192] 420 -> 271
[8240,1,256] k [11,1,256,256] 165 -> 108
[19,19,256,16] k [4,4,256,4096] 1672 -> 1076

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

b12499

b12499 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 18 Nov 05:23
401563f

b12498

b12498 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 18 Nov 04:18
5297534

server: slot_sched: fix lora and recurrent (#32434)

  • server: fix lora and recurrent

  • tests: run the hybrid SWA checkpoint test with one token per ubatch

With and without the cache, the prompt was not split into the same
ubatches, and the generated tokens are decoded one at a time while the
uncached run processes them in a batch. On CUDA this moves the top
logprobs of the tiny model by up to 0.03 and swaps near ties, so the
test failed while the cache behaved as expected.

With one token per ubatch both runs go through the same kernels and
the output is compared exactly.


Co-authored-by: Pascal admin@serveurperso.com

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

b12497

b12497 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 18 Nov 04:02
3574df8

b12496

b12496 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 18 Nov 03:44
2e53a0b

b12494

b12494 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 18 Nov 02:37
307e3f1

mtmd: add kokoro tts support (#32413)

  • jinja: impl scope-chain key lookup

  • add test for call_statement lambda

  • chat: add tts_dict for kokoro G2P template

  • more testing

  • value_float_t range check

  • no json-in-meta, add llama_model_meta_arr_str/_n

  • mtmd: add kokoro tts support

  • detailed guide about G2P and text norm port

  • refine G2P

  • clean up

  • migrate batch api

  • nits

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

b12493

b12493 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 18 Nov 02:21
9048bad

cuda: use the implicit GEMM conv2d path for F32 kernels (#32482)

The fast CONV_2D path (implicit GEMM with MMA, or cuBLAS for
pointwise convolutions) required an F16 kernel. F32 kernels fell
back to the naive direct convolution, which is much slower than
im2col + mul_mat on large channel counts.

Convert F32 kernels to F16 in a pool buffer, as is already done for
the input, and take the same path with F32 accumulation. The Metal
implicit GEMM kernel already stages both operands as F16.

Website:

Attestations:

macOS/iOS:

Linux:

Android:

Windows:

openEuler:

  • DISABLED
  • openEuler x86 (310p)
  • openEuler x86 (910b, ACL Graph)
  • openEuler aarch64 (310p)
  • openEuler aarch64 (910b, ACL Graph)

UI:

b12492

b12492 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 18 Nov 02:00
a9d43ec

b12491

b12491 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 18 Nov 01:35
8c16bab

b12490

b12490 Pre-release
Pre-release

Choose a tag to compare

@github-actions github-actions released this 18 Nov 01:14
ad9d9e8