Skip to content

Commit

Permalink
Simplify and rewrite reference packm kernels. (#610)
Browse files Browse the repository at this point in the history 
Details:
- Reorganized the way kernels are stored within the cntx_t structure so
  that rather than having a function pointer for every supported size of
  unrolled packm kernel (2xk, 3xk, 4xk, etc.), we store only two packm
  kernels per datatype: one to pack MRxk micropanels and one to pack
  NRxk micropanels.
  - NOTE: The "bb" (broadcast B) reference kernels have been merged into
    the "standard" kernels (packm [including 1er and unpackm], gemm,
    trsm, gemmtrsm). This replication factor is controlled by
    BLIS_BB[MN]_[sdcz] etc. Power9/10 needs testing since only a
    replication factor of 1 has been tested. armsve also needs testing
    since the MR value isn't available as a macro.
- Simplified the bli_cntx_*() APIs to conform to the new unified kernel
  array within the cntx_t. Updated existing bli_cntx_init_<subconfig>()
  function definitions for all subconfigurations.
- Consolidated all kernel id types (e.g. l1vkr_t, l1mkr_t, l3ukr_t,
  etc.) into one kernel id type: ukr_t.
- Various edits, updates, and rewrites of reference kernels pursuant to
  the aforementioned changes.
- Define compile-time macro constants (BLIS_MR_[sdcz], BLIS_NR_[sdcz],
  and friends) in bli_kernel_macro_defs.h, but only when the macro
  BLIS_IN_REF_KERNEL is defined by the build system.
- Loose ends:
  - Still need to update documentation, including:
    - docs/ConfigurationHowTo.md
    - docs/KernelsHowTo.md
    to reflect changes made in this commit.
- (cherry picked from commit ae10d94)

Fixed type bug in bli_cntx_set_ukr_prefs().

Details:
- Fixed a bug in bli_cntx_set_ukr_prefs() which erroneously typecast the
  num_t value read from va_args() down to a bool before being stored
  within the cntx_t. This bug was introduced on April 6th 2022, in
  ae10d94. This caused the ukernel preferences for double real and
  double complex to go unchanged while the preferences for single real
  and single complex were corrupted by the former datatypes'
  preference values. The bug manifested as degraded performance for
  subconfigurations that registered column-preferential ukernels. The
  reason is that the erroneous preferences trigger unnecessary
  transpositions in the operation, which forces the gemm ukernel to
  compute on matrices that are not stored according to its preference.
  Thanks to Devin Matthews, Jeff Diamond, and Leick Robinson for their
  extensive efforts and assistance in tracking down this issue.
- Augmented the informational header that is output by the testsuite to
  include ukernel preferences for gemm, gemmtrsm_[lu], and trsm_[lu].
- CREDITS file update.
- (cherry picked from commit 667f201)
  • Loading branch information
@fgvanzee
fgvanzee committed Aug 7, 2023
1 parent 59e6151 commit 3c652ee
Show file tree
Hide file tree
Showing 220 changed files with 5,367 additions and 10,290 deletions.
1 change: 1 addition & 0 deletions CREDITS
View file
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ but many others have contributed code and feedback, including
Michael Rader @mrader1248
Pradeep Rao @pradeeptrgit (AMD)
Aleksei Rechinskii
Leick Robinson @LeickR (Oracle)
Karl Rupp @karlrupp
Paul Sandoz @PaulSandoz (Oracle)
Martin Schatz (The University of Texas at Austin)
Expand Down
8 changes: 4 additions & 4 deletions addon/gemmd/attic/bao_gemmd_bp_var2.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,7 @@ void PASTECH2(bao_,ch,varname) \
function pointer type. */ \
/*
PASTECH(ch,gemm_ukr_ft) \
gemm_ukr = bli_cntx_get_l3_nat_ukr_dt( dt, BLIS_GEMM_UKR, cntx ); \
gemm_ukr = bli_cntx_get_ukr_dt( dt, BLIS_GEMM_UKR, cntx ); \
*/ \
\
/* Temporary C buffer for edge cases. Note that the strides of this
Expand All @@ -175,7 +175,7 @@ void PASTECH2(bao_,ch,varname) \
ctype ct[ BLIS_STACK_BUF_MAX_SIZE \
/ sizeof( ctype ) ] \
__attribute__((aligned(BLIS_STACK_BUF_ALIGN_SIZE))); \
const bool col_pref = bli_cntx_l3_nat_ukr_prefers_cols_dt( dt, BLIS_GEMM_UKR, cntx ); \
const bool col_pref = bli_cntx_ukr_prefers_cols_dt( dt, BLIS_GEMM_UKR, cntx ); \
const inc_t rs_ct = ( col_pref ? 1 : NR ); \
const inc_t cs_ct = ( col_pref ? MR : 1 ); \
*/ \
Expand Down Expand Up @@ -536,7 +536,7 @@ void PASTECH2(bao_,ch,varname) \
/* Query the context for the microkernel address and cast it to its
function pointer type. */ \
PASTECH(ch,gemm_ukr_ft) \
gemm_ukr = bli_cntx_get_l3_nat_ukr_dt( dt, BLIS_GEMM_UKR, cntx ); \
gemm_ukr = bli_cntx_get_ukr_dt( dt, BLIS_GEMM_UKR, cntx ); \
\
/* Temporary C buffer for edge cases. Note that the strides of this
temporary buffer are set so that they match the storage of the
Expand All @@ -545,7 +545,7 @@ void PASTECH2(bao_,ch,varname) \
ctype ct[ BLIS_STACK_BUF_MAX_SIZE \
/ sizeof( ctype ) ] \
__attribute__((aligned(BLIS_STACK_BUF_ALIGN_SIZE))); \
const bool col_pref = bli_cntx_l3_nat_ukr_prefers_cols_dt( dt, BLIS_GEMM_UKR, cntx ); \
const bool col_pref = bli_cntx_ukr_prefers_cols_dt( dt, BLIS_GEMM_UKR, cntx ); \
const inc_t rs_ct = ( col_pref ? 1 : NR ); \
const inc_t cs_ct = ( col_pref ? MR : 1 ); \
\
Expand Down
2 changes: 1 addition & 1 deletion addon/gemmd/bao_gemmd.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ void bao_gemmd_ex
// contiguous columns, or if C is stored by columns and the micro-kernel
// prefers contiguous rows, transpose the entire operation to allow the
// micro-kernel to access elements of C in its preferred manner.
if ( bli_cntx_l3_vir_ukr_dislikes_storage_of( &c_local, BLIS_GEMM_UKR, cntx ) )
if ( bli_cntx_dislikes_storage_of( &c_local, BLIS_GEMM_VIR_UKR, cntx ) )
{
bli_obj_swap( &a_local, &b_local );

Expand Down
2 changes: 1 addition & 1 deletion addon/gemmd/bao_gemmd_bp_var1.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -163,7 +163,7 @@ void PASTECH2(bao_,ch,varname) \
/* Query the context for the microkernel address and cast it to its
function pointer type. */ \
PASTECH(ch,gemm_ukr_ft) \
gemm_ukr = bli_cntx_get_l3_nat_ukr_dt( dt, BLIS_GEMM_UKR, cntx ); \
gemm_ukr = bli_cntx_get_ukr_dt( dt, BLIS_GEMM_UKR, cntx ); \
\
/* Compute partitioning step values for each matrix of each loop. */ \
const inc_t jcstep_c = cs_c; \
Expand Down
6 changes: 3 additions & 3 deletions addon/gemmd/bao_packm_cxk.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,15 +55,15 @@ void PASTECH2(bao_,ch,opname) \
/* Note that we use panel_dim_max, not panel_dim, to query the packm
kernel function pointer. This means that we always use the same
kernel, even for edge cases. */ \
num_t dt = PASTEMAC(ch,type); \
l1mkr_t ker_id = panel_dim_max; \
num_t dt = PASTEMAC(ch,type); \
ukr_t ker_id = bli_is_col_packed( schema ) ? BLIS_PACKM_NRXK_KER : BLIS_PACKM_MRXK_KER; \
\
PASTECH2(ch,opname,_ker_ft) f; \
\
/* Query the context for the packm kernel corresponding to the current
panel dimension, or kernel id. If the id is invalid, the function will
return NULL. */ \
f = bli_cntx_get_packm_ker_dt( dt, ker_id, cntx ); \
f = bli_cntx_get_ukr_dt( dt, ker_id, cntx ); \
\
/* If there exists a kernel implementation for the micro-panel dimension
provided, we invoke the implementation. Otherwise, we use scal2m. */ \
Expand Down
4 changes: 4 additions & 0 deletions common.mk
View file
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,8 @@ get-refinit-cflags-for = $(strip $(call load-var-for,COPTFLAGS,$(1)) \
-DBLIS_CNAME=$(1) \
$(BUILD_CPPFLAGS) \
$(BUILD_SYMFLAGS) \
-DBLIS_IN_REF_KERNEL=1 \
-include $(CONFIG_PATH)/$(1)/bli_kernel_defs_$(1).h \
)

get-refkern-cflags-for = $(strip $(call load-var-for,CROPTFLAGS,$(1)) \
Expand All @@ -129,6 +131,8 @@ get-refkern-cflags-for = $(strip $(call load-var-for,CROPTFLAGS,$(1)) \
-DBLIS_CNAME=$(1) \
$(BUILD_CPPFLAGS) \
$(BUILD_SYMFLAGS) \
-DBLIS_IN_REF_KERNEL=1 \
-include $(CONFIG_PATH)/$(1)/bli_kernel_defs_$(1).h \
)

get-config-cflags-for = $(strip $(call load-var-for,COPTFLAGS,$(1)) \
Expand Down
100 changes: 30 additions & 70 deletions config/a64fx/bli_cntx_init_a64fx.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,34 +38,42 @@
void bli_cntx_init_a64fx( cntx_t* cntx )
{
blksz_t blkszs[ BLIS_NUM_BLKSZS ];
blksz_t thresh[ BLIS_NUM_THRESH ];

// Set default kernel blocksizes and functions.
bli_cntx_init_a64fx_ref( cntx );

// -------------------------------------------------------------------------

// Update the context with optimized native gemm micro-kernels and
// their storage preferences.
bli_cntx_set_l3_nat_ukrs
// Update the context with optimized native gemm micro-kernels.
bli_cntx_set_ukrs
(
4,
BLIS_GEMM_UKR, BLIS_FLOAT, bli_sgemm_armsve_asm_2vx10_unindexed, FALSE,
BLIS_GEMM_UKR, BLIS_DOUBLE, bli_dgemm_armsve_asm_2vx10_unindexed, FALSE,
BLIS_GEMM_UKR, BLIS_SCOMPLEX, bli_cgemm_armsve_asm_2vx10_unindexed, FALSE,
BLIS_GEMM_UKR, BLIS_DCOMPLEX, bli_zgemm_armsve_asm_2vx10_unindexed, FALSE,
cntx
cntx,

// level-3
BLIS_GEMM_UKR, BLIS_FLOAT, bli_sgemm_armsve_asm_2vx10_unindexed,
BLIS_GEMM_UKR, BLIS_DOUBLE, bli_dgemm_armsve_asm_2vx10_unindexed,
BLIS_GEMM_UKR, BLIS_SCOMPLEX, bli_cgemm_armsve_asm_2vx10_unindexed,
BLIS_GEMM_UKR, BLIS_DCOMPLEX, bli_zgemm_armsve_asm_2vx10_unindexed,

// packm
BLIS_PACKM_MRXK_KER, BLIS_DOUBLE, bli_dpackm_armsve512_asm_16xk,
BLIS_PACKM_NRXK_KER, BLIS_DOUBLE, bli_dpackm_armsve512_asm_10xk,

BLIS_VA_END
);

// Set SVE-512 packing routine.
bli_cntx_set_packm_kers
// Update the context with storage preferences.
bli_cntx_set_ukr_prefs
(
2,
BLIS_PACKM_10XK_KER, BLIS_DOUBLE, bli_dpackm_armsve512_asm_10xk,
// 12xk is not used and disabled for GCC 8-9 compatibility.
// BLIS_PACKM_12XK_KER, BLIS_DOUBLE, bli_dpackm_armsve512_int_12xk,
BLIS_PACKM_16XK_KER, BLIS_DOUBLE, bli_dpackm_armsve512_asm_16xk,
cntx
cntx,

// level-3
BLIS_GEMM_UKR_ROW_PREF, BLIS_FLOAT, FALSE,
BLIS_GEMM_UKR_ROW_PREF, BLIS_DOUBLE, FALSE,
BLIS_GEMM_UKR_ROW_PREF, BLIS_SCOMPLEX, FALSE,
BLIS_GEMM_UKR_ROW_PREF, BLIS_DCOMPLEX, FALSE,

BLIS_VA_END
);

// Initialize level-3 blocksize objects with architecture-specific values.
Expand All @@ -80,66 +88,18 @@ void bli_cntx_init_a64fx( cntx_t* cntx )
// blocksizes (and multiples) for native execution.
bli_cntx_set_blkszs
(
BLIS_NAT, 5,
cntx,

// level-3
BLIS_NC, &blkszs[ BLIS_NC ], BLIS_NR,
BLIS_KC, &blkszs[ BLIS_KC ], BLIS_KR,
BLIS_MC, &blkszs[ BLIS_MC ], BLIS_MR,
BLIS_NR, &blkszs[ BLIS_NR ], BLIS_NR,
BLIS_MR, &blkszs[ BLIS_MR ], BLIS_MR,
cntx
);

#if 0
// Initialize sup thresholds with architecture-appropriate values.
// s d c z
bli_blksz_init_easy( &thresh[ BLIS_MT ], -1, 65, -1, -1 );
bli_blksz_init_easy( &thresh[ BLIS_NT ], -1, 65, -1, -1 );
bli_blksz_init_easy( &thresh[ BLIS_KT ], -1, 65, -1, -1 );

// Initialize the context with the sup thresholds.
bli_cntx_set_l3_sup_thresh
(
3,
BLIS_MT, &thresh[ BLIS_MT ],
BLIS_NT, &thresh[ BLIS_NT ],
BLIS_KT, &thresh[ BLIS_KT ],
cntx
BLIS_VA_END
);

// Update the context with optimized small/unpacked gemm kernels.
bli_cntx_set_l3_sup_kers
(
4,
BLIS_RRR, BLIS_DOUBLE, bli_dgemmsup_rv_armsve_10x2v_unindexed, TRUE,
BLIS_RCR, BLIS_DOUBLE, bli_dgemmsup_rv_armsve_10x2v_unindexed, TRUE,
BLIS_CCR, BLIS_DOUBLE, bli_dgemmsup_rv_armsve_10x2v_unindexed, TRUE,
BLIS_CCC, BLIS_DOUBLE, bli_dgemmsup_rv_armsve_10x2v_unindexed, TRUE,
cntx
);

// Initialize level-3 sup blocksize objects with architecture-specific
// values.
// s d c z
bli_blksz_init_easy( &blkszs[ BLIS_MR ], -1, 10, -1, -1 );
bli_blksz_init_easy( &blkszs[ BLIS_NR ], -1, 16, -1, -1 );
bli_blksz_init_easy( &blkszs[ BLIS_MC ], -1, 120, -1, -1 );
bli_blksz_init_easy( &blkszs[ BLIS_KC ], -1, 256, -1, -1 );
bli_blksz_init_easy( &blkszs[ BLIS_NC ], -1, 4080, -1, -1 );

// Update the context with the current architecture's register and cache
// blocksizes for small/unpacked level-3 problems.
bli_cntx_set_l3_sup_blkszs
(
5,
BLIS_NC, &blkszs[ BLIS_NC ],
BLIS_KC, &blkszs[ BLIS_KC ],
BLIS_MC, &blkszs[ BLIS_MC ],
BLIS_NR, &blkszs[ BLIS_NR ],
BLIS_MR, &blkszs[ BLIS_MR ],
cntx
);
#endif

// Set A64FX cache sector sizes for each PE/CMG
// SC Fugaku might disable users' setting cache sizes.
#if !defined(CACHE_SECTOR_SIZE_READONLY)
Expand Down
52 changes: 52 additions & 0 deletions config/a64fx/bli_kernel_defs_a64fx.h
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
/*
BLIS
An object-based framework for developing high-performance BLAS-like
libraries.
Copyright (C) 2022, The University of Texas at Austin
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
- Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
- Neither the name(s) of the copyright holder(s) nor the names of its
contributors may be used to endorse or promote products derived
from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

//#ifndef BLIS_KERNEL_DEFS_H
//#define BLIS_KERNEL_DEFS_H


// -- REGISTER BLOCK SIZES (FOR REFERENCE KERNELS) ----------------------------

#define BLIS_MR_s 32
#define BLIS_MR_d 16
#define BLIS_MR_c 16
#define BLIS_MR_z 8

#define BLIS_NR_s 10
#define BLIS_NR_d 10
#define BLIS_NR_c 10
#define BLIS_NR_z 10

//#endif

29 changes: 22 additions & 7 deletions config/altra/bli_cntx_init_altra.c
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,25 @@ void bli_cntx_init_altra( cntx_t* cntx )

// Update the context with optimized native gemm micro-kernels and
// their storage preferences.
bli_cntx_set_l3_nat_ukrs
bli_cntx_set_ukrs
(
2,
BLIS_GEMM_UKR, BLIS_FLOAT, bli_sgemm_armv8a_asm_8x12, FALSE,
BLIS_GEMM_UKR, BLIS_DOUBLE, bli_dgemm_armv8a_asm_6x8, FALSE,
cntx
cntx,

BLIS_GEMM_UKR, BLIS_FLOAT, bli_sgemm_armv8a_asm_8x12,
BLIS_GEMM_UKR, BLIS_DOUBLE, bli_dgemm_armv8a_asm_6x8,

BLIS_VA_END
);

// Update the context with storage preferences.
bli_cntx_set_ukr_prefs
(
cntx,

BLIS_GEMM_UKR_ROW_PREF, BLIS_FLOAT, FALSE,
BLIS_GEMM_UKR_ROW_PREF, BLIS_DOUBLE, FALSE,

BLIS_VA_END
);

// Initialize level-3 blocksize objects with architecture-specific values.
Expand All @@ -66,13 +79,15 @@ void bli_cntx_init_altra( cntx_t* cntx )
// blocksizes (and multiples) for native execution.
bli_cntx_set_blkszs
(
BLIS_NAT, 5,
cntx,

BLIS_NC, &blkszs[ BLIS_NC ], BLIS_NR,
BLIS_KC, &blkszs[ BLIS_KC ], BLIS_KR,
BLIS_MC, &blkszs[ BLIS_MC ], BLIS_MR,
BLIS_NR, &blkszs[ BLIS_NR ], BLIS_NR,
BLIS_MR, &blkszs[ BLIS_MR ], BLIS_MR,
cntx

BLIS_VA_END
);
}

48 changes: 48 additions & 0 deletions config/altra/bli_kernel_defs_altra.h
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
/*
BLIS
An object-based framework for developing high-performance BLAS-like
libraries.
Copyright (C) 2022, The University of Texas at Austin
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
- Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
- Neither the name(s) of the copyright holder(s) nor the names of its
contributors may be used to endorse or promote products derived
from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*/

//#ifndef BLIS_KERNEL_DEFS_H
//#define BLIS_KERNEL_DEFS_H


// -- REGISTER BLOCK SIZES (FOR REFERENCE KERNELS) ----------------------------

#define BLIS_MR_s 8
#define BLIS_MR_d 6

#define BLIS_NR_s 12
#define BLIS_NR_d 8

//#endif

Loading

0 comments on commit 3c652ee

Please sign in to comment.