This patch adds 'bitlock_t' type and bit_lock() and
bit_unlock() to support locking/release functionality
based on individual bit position. These functions use
atomic bit set and clear instructions which require
FEAT_LSE mandatory from Armv8.1.
Change-Id: I3eb0f29bbccefe6c0f69061aa701187a6364df0c
Signed-off-by: AlexeiFedorov <Alexei.Fedorov@arm.com>
Align entire TF-A to use Arm in copyright header.
Change-Id: Ief9992169efdab61d0da6bd8c5180de7a4bc2244
Signed-off-by: Govindraj Raja <govindraj.raja@arm.com>
is_lock_acquired() function is only used in assert() statements, so when
compiling without asserts, e.g. with DEBUG=0, the function is unused.
this is okay when compiling with gcc because the function is marked as
inline but that doesn't work for clang. let's mark this as __unused to
avoid -Wunused-function warning-as-error.
Change-Id: I93f808fd15f715a65d1bd4f7592affb7997c4bad
Signed-off-by: Okash Khawaja <okash@google.com>
bakery_lock_normal.c uses the raw register accessor, read_sctlr(_el3)
to check whether the dcache is enabled.
Using is_dcache_enabled() is cleaner, and a good abstraction for
the library code like this.
A problem is is_dcache_enabled() is declared in the local header,
lib/xlat_tables_v2/xlat_tables_private.h
I searched for a good place to declare this helper. Moving it to
arch_helpers.h, closed to cache operation helpers, looks good enough
to me.
I also changed the type of 'is_cached' to bool for consistency,
and to avoid MISRA warnings.
Change-Id: I9b016f67bc8eade25c316aa9c0db0fa4cd375b79
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
ARM has a weak memory ordering model. This means that without
explicit barriers, memory accesses can be observed differently
than program order. In this case, the cache invalidate instruction
can be observed after the subsequent read to address.
To solve this, a DMB instruction is required between the cache
invalidate and the read. This ensures that the cache invalidate
completes before all memory accesses in program order after the DMB.
This patch updates the 'read_cache_op' macro to issue a DMB after
the cache invalidate instruction to fix this anomaly.
Change-Id: Iac9a90d228c57ba8bcdca7e409ea6719546ab441
Signed-off-by: Varun Wadekar <vwadekar@nvidia.com>
bakery_lock_get() uses DMB LD after lock acquisition and
bakery_lock_release() uses DMB ST before releasing the lock. This is
insufficient in both cases. With just DMB LD, stores in the critical
section can be reordered before the DMB LD which could mean writes in
the critical section completing before the lock has been acquired
successfully. Similarly, with just DMB ST, a load in the critical section
could be reordered after the the DMB ST. DMB is the least expensive
barrier that can provide the required ordering.
Signed-off-by: Raghu Krishnamurthy <raghu.ncstate@icloud.com>
Change-Id: Ieb74cbf5b76b09e1789331b71f37f7c660221b0e
Make the spinlock implementation use ARMv8.1-LSE CAS instruction based
on a platform build option. The CAS-based implementation used to be
unconditionally selected for all ARM8.1+ platforms.
The previous CAS spinlock implementation had a bug wherein the spin_unlock()
implementation had an `sev` after `stlr` which is not sufficient. A dsb is
needed to ensure that the stlr completes prior to the sev. Having a dsb is
heavyweight and a better solution would be to use load exclusive semantics
to monitor the lock and wake up from wfe when a store happens to the lock.
The patch implements the same.
Change-Id: I5283ce4a889376e4cc01d1b9d09afa8229a2e522
Signed-off-by: Soby Mathew <soby.mathew@arm.com>
Signed-off-by: Olivier Deprez <olivier.deprez@arm.com>
NOTE: AARCH32/AARCH64 macros are now deprecated in favor of __aarch64__.
All common C compilers pre-define the same macros to signal which
architecture the code is being compiled for: __arm__ for AArch32 (or
earlier versions) and __aarch64__ for AArch64. There's no need for TF-A
to define its own custom macros for this. In order to unify code with
the export headers (which use __aarch64__ to avoid another dependency),
let's deprecate the AARCH32 and AARCH64 macros and switch the code base
over to the pre-defined standard macro. (Since it is somewhat
unintuitive that __arm__ only means AArch32, let's standardize on only
using __aarch64__.)
Change-Id: Ic77de4b052297d77f38fc95f95f65a8ee70cf200
Signed-off-by: Julius Werner <jwerner@chromium.org>
This patch removes .arch "arm8.1-a" and "armv8-a"
directives which overwrite ASFLAGS_aarch64 option based
on ARM_ARCH_MINOR passed to Makefile and cause
translation errors like
"selected processor does not support `bti jc'"
for armv8.5-a targets when BTI support is enabled.
Change-Id: Idca5b66ed1e5d86e2188b0c0f16c3819990957c4
Signed-off-by: Alexei Fedorov <Alexei.Fedorov@arm.com>
This patch introduces explicit linker variables to mark the start and
end of the per-cpu bakery lock section to help bakery_lock_normal.c
calculate the size of the section. This patch removes the previously
used '__PERCPU_BAKERY_LOCK_SIZE__' linker variable to make the code
uniform across GNU linker and ARM linker.
Change-Id: Ie0c51702cbc0fe8a2076005344a1fcebb48e7cca
Signed-off-by: Varun Wadekar <vwadekar@nvidia.com>
Enforce full include path for includes. Deprecate old paths.
The following folders inside include/lib have been left unchanged:
- include/lib/cpus/${ARCH}
- include/lib/el3_runtime/${ARCH}
The reason for this change is that having a global namespace for
includes isn't a good idea. It defeats one of the advantages of having
folders and it introduces problems that are sometimes subtle (because
you may not know the header you are actually including if there are two
of them).
For example, this patch had to be created because two headers were
called the same way: e0ea0928d5 ("Fix gpio includes of mt8173 platform
to avoid collision."). More recently, this patch has had similar
problems: 46f9b2c3a2 ("drivers: add tzc380 support").
This problem was introduced in commit 4ecca33988 ("Move include and
source files to logical locations"). At that time, there weren't too
many headers so it wasn't a real issue. However, time has shown that
this creates problems.
Platforms that want to preserve the way they include headers may add the
removed paths to PLAT_INCLUDES, but this is discouraged.
Change-Id: I39dc53ed98f9e297a5966e723d1936d6ccf2fc8f
Signed-off-by: Antonio Nino Diaz <antonio.ninodiaz@arm.com>
With the current implementation, it's possible for a contender to
observe accesses in the Critical Section before acquiring or releasing
the lock. Insert fencing in the locking and release codes to prevent any
reorder.
FixesARM-software/tf-issues#609
Change-Id: I773b82aa41dd544a2d3dbacb9a4b42c9eb767bbb
Signed-off-by: Jeenu Viswambharan <jeenu.viswambharan@arm.com>
Use a _ prefix for macro arguments to prevent that argument from
hiding variables of the same name in the outer scope
Rule 5.3: An identifier declared in an inner scope shall not
hide an identifier declared in an outer scope
Fixed For:
make PLAT=fvp USE_COHERENT_MEM=0
Change-Id: If50c583d3b63799ee6852626b15be00c0f6b10a0
Signed-off-by: Daniel Boulby <daniel.boulby@arm.com>
Use a _ prefix for Macro arguments to prevent that argument from
hiding variables of the same name in the outer scope
Rule 5.3: An identifier declared in an inner scope shall not
hide an identifier declared in an outer scope
Fixed For:
make LOG_LEVEL=50 PLAT=fvp
Change-Id: I67b6b05cbad4aeca65ce52981b4679b340604708
Signed-off-by: Daniel Boulby <daniel.boulby@arm.com>
Void pointers have been used to access linker symbols, by declaring an
extern pointer, then taking the address of it. This limits symbols
values to aligned pointer values. To remove this restriction an
IMPORT_SYM macro has been introduced, which declares it as a char
pointer and casts it to the required type.
Change-Id: I89877fc3b13ed311817bb8ba79d4872b89bfd3b0
Signed-off-by: Joel Hutton <Joel.Hutton@Arm.com>
The macro concisely expresses and requires architecture version to be at
least as required by its arguments. This would be useful when extending
Trusted Firmware functionality for future architecture revisions.
Replace similar usage in the current code base with the new macro.
Change-Id: I9dcd0aa71a663eabd02ed9632b8ce87611fa5a57
Signed-off-by: Jeenu Viswambharan <jeenu.viswambharan@arm.com>
To make software license auditing simpler, use SPDX[0] license
identifiers instead of duplicating the license text in every file.
NOTE: Files that have been imported by FreeBSD have not been modified.
[0]: https://spdx.org/
Change-Id: I80a00e1f641b8cc075ca5a95b10607ed9ed8761a
Signed-off-by: dp-arm <dimitris.papastamos@arm.com>
This patch fixes a compilation issue with bakery locks when
PSCI library is compiled with USE_COHERENT_MEM = 0 build option.
Change-Id: Ic7f6cf9f2bb37f8a946eafbee9cbc3bf0dc7e900
Signed-off-by: Soby Mathew <soby.mathew@arm.com>
The ARMv8v.1 architecture extension has introduced support for far
atomics, which includes compare-and-swap. Compare and Swap instruction
is only available for AArch64.
Introduce build options to choose the architecture versions to target
ARM Trusted Firmware:
- ARM_ARCH_MAJOR: selects the major version of target ARM
Architecture. Default value is 8.
- ARM_ARCH_MINOR: selects the minor version of target ARM
Architecture. Default value is 0.
When:
(ARM_ARCH_MAJOR > 8) || ((ARM_ARCH_MAJOR == 8) && (ARM_ARCH_MINOR >= 1)),
for AArch64, Compare and Swap instruction is used to implement spin
locks. Otherwise, the implementation falls back to using
load-/store-exclusive instructions.
Update user guide, and introduce a section in Firmware Design guide to
summarize support for features introduced in ARMv8 Architecture
Extensions.
Change-Id: I73096a0039502f7aef9ec6ab3ae36680da033f16
Signed-off-by: Jeenu Viswambharan <jeenu.viswambharan@arm.com>
This patch fixes an issue in the normal memory bakery lock
implementation. During assertion of lock status, there is a possibility
that the assertion could fail. This is because the previous update done
to the lock status by the owning CPU when not participating in cache
coherency could result in stale data in the cache due to cache maintenance
operations not propagating to all the caches. This patch fixes this issue
by doing an extra read cache maintenance operation prior to the assertion.
FixesARM-software/tf-issues#402
Change-Id: I0f38a7c52476a4f58e17ebe0141d256d198be88d
Signed-off-by: Soby Mathew <soby.mathew@arm.com>
This patch adds AArch32 support to cpu ops, context management,
per-cpu data and spinlock libraries. The `entrypoint_info`
structure is modified to add support for AArch32 register
arguments. The CPU operations for AEM generic cpu in AArch32
mode is also added.
Change-Id: I1e52e79f498661d8f31f1e7b3a29e222bc7a4483
This patch moves the assembly exclusive lock library code
`spinlock.S` into architecture specific folder `aarch64`.
A stub file which includes the file from new location is
retained at the original location for compatibility. The BL
makefiles are also modified to include the file from the new
location.
Change-Id: Ide0b601b79c439e390c3a017d93220a66be73543
This patch reworks type usage in generic code, drivers and ARM platform files
to make it more portable. The major changes done with respect to
type usage are as listed below:
* Use uintptr_t for storing address instead of uint64_t or unsigned long.
* Review usage of unsigned long as it can no longer be assumed to be 64 bit.
* Use u_register_t for register values whose width varies depending on
whether AArch64 or AArch32.
* Use generic C types where-ever possible.
In addition to the above changes, this patch also modifies format specifiers
in print invocations so that they are AArch64/AArch32 agnostic. Only files
related to upcoming feature development have been reworked.
Change-Id: I9f8c78347c5a52ba7027ff389791f1dad63ee5f8
This patch unifies the bakery lock api's across coherent and normal
memory implementation of locks by using same data type `bakery_lock_t`
and similar arguments to functions.
A separate section `bakery_lock` has been created and used to allocate
memory for bakery locks using `DEFINE_BAKERY_LOCK`. When locks are
allocated in normal memory, each lock for a core has to spread
across multiple cache lines. By using the total size allocated in a
separate cache line for a single core at compile time, the memory for
other core locks is allocated at link time by multiplying the single
core locks size with (PLATFORM_CORE_COUNT - 1). The normal memory lock
algorithm now uses lock address instead of the `id` in the per_cpu_data.
For locks allocated in coherent memory, it moves locks from
tzfw_coherent_memory to bakery_lock section.
The bakery locks are allocated as part of bss or in coherent memory
depending on usage of coherent memory. Both these regions are
initialised to zero as part of run_time_init before locks are used.
Hence, bakery_lock_init() is made an empty function as the lock memory
is already initialised to zero.
The above design lead to the removal of psci bakery locks from
non_cpu_power_pd_node to psci_locks.
NOTE: THE BAKERY LOCK API WHEN USE_COHERENT_MEM IS NOT SET HAS CHANGED.
THIS IS A BREAKING CHANGE FOR ALL PLATFORM PORTS THAT ALLOCATE BAKERY
LOCKS IN NORMAL MEMORY.
Change-Id: Ic3751c0066b8032dcbf9d88f1d4dc73d15f61d8b
This patch migrates the rest of Trusted Firmware excluding Secure Payload and
the dispatchers to the new platform and context management API. The per-cpu
data framework APIs which took MPIDRs as their arguments are deleted and only
the ones which take core index as parameter are retained.
Change-Id: I839d05ad995df34d2163a1cfed6baa768a5a595d
In order for the symbol table in the ELF file to contain the size of
functions written in assembly, it is necessary to report it to the
assembler using the .size directive.
To fulfil the above requirements, this patch introduces an 'endfunc'
macro which contains the .endfunc and .size directives. It also adds
a .func directive to the 'func' assembler macro.
The .func/.endfunc have been used so the assembler can fail if
endfunc is omitted.
FixesARM-Software/tf-issues#295
Change-Id: If8cb331b03d7f38fe7e3694d4de26f1075b278fc
Signed-off-by: Kévin Petit <kevin.petit@arm.com>
This patch removes the `owner` field in bakery_lock_t structure which
is the data structure used in the bakery lock implementation that uses
coherent memory. The assertions to protect against recursive lock
acquisition were based on the 'owner' field. They are now done based
on the bakery lock ticket number. These assertions are also added
to the bakery lock implementation that uses normal memory as well.
Change-Id: If4850a00dffd3977e218c0f0a8d145808f36b470
This patch optimizes the data structure used with the bakery lock
implementation for coherent memory to save memory and minimize memory
accesses. These optimizations were already part of the bakery lock
implementation for normal memory and this patch now implements
it for the coherent memory implementation as well. Also
included in the patch is a cleanup to use the do-while loop while
waiting for other contenders to finish choosing their tickets.
Change-Id: Iedb305473133dc8f12126726d8329b67888b70f1
This patch moves the bakery locks out of coherent memory to normal memory.
This implies that the lock information needs to be placed on a separate cache
line for each cpu. Hence the bakery_lock_info_t structure is allocated in the
per-cpu data so as to minimize memory wastage. A similar platform per-cpu
data is introduced for the platform locks.
As a result of the above changes, the bakery lock api is completely changed.
Earlier, a reference to the lock structure was passed to the lock implementation.
Now a unique-id (essentially an index into the per-cpu data array) and an offset
into the per-cpu data for bakery_info_t needs to be passed to the lock
implementation.
Change-Id: I1e76216277448713c6c98b4c2de4fb54198b39e0
This patch is an optimization in the bakery_lock_get() function
which removes the wfe() when waiting for other contenders to choose
their ticket i.e when their `entering` flag is set. Since the time
taken to execute bakery_get_ticket() by other contenders is bounded,
this wait is a bounded time wait. Hence the removal of wfe() and the
corresponding sev() and dsb() in bakery_get_ticket() may result
in better time performance during lock acquisition.
Change-Id: I141bb21294226b54cb6e89e7cac0175c553afd8d
This patch fixes a bug in the bakery lock implementation where a data
synchronisation barrier instruction is not issued before sending an event as
mandated by the ARMv8 ARM. This can cause a event to be signalled before the
related memory accesses have completed resulting in erroneous execution.
FixesARM-software/tf-issues#272
Change-Id: I5ce02bf70afb001d967b9fa4c3f77442931d5349
The bakery lock code currently expects the calling code to pass
the MPIDR_EL1 of the current CPU.
This is not always done correctly. Also the change to provide
inline access to system registers makes it more efficient for the
bakery lock code to obtain the MPIDR_EL1 directly.
This change removes the mpidr parameter from the bakery lock
interface, and results in a code reduction of 160 bytes for the
ARM FVP port.
FixesARM-software/tf-issues#213
Change-Id: I7ec7bd117bcc9794a0d948990fcf3336a367d543
Previously, platform.h contained many declarations and definitions
used for different purposes. This file has been split so that:
* Platform definitions used by common code that must be defined
by the platform are now in platform_def.h. The exact include
path is exported through $PLAT_INCLUDES in the platform makefile.
* Platform definitions specific to the FVP platform are now in
/plat/fvp/fvp_def.h.
* Platform API declarations specific to the FVP platform are now
in /plat/fvp/fvp_private.h.
* The remaining platform API declarations that must be ported by
each platform are still in platform.h but this file has been
moved to /include/plat/common since this can be shared by all
platforms.
Change-Id: Ieb3bb22fbab3ee8027413c6b39a783534aee474a
Reduce the number of header files included from other header
files as much as possible without splitting the files. Use forward
declarations where possible. This allows removal of some unnecessary
"#ifndef __ASSEMBLY__" statements.
Also, review the .c and .S files for which header files really need
including and reorder the #include statements alphabetically.
FixesARM-software/tf-issues#31
Change-Id: Iec92fb976334c77453e010b60bcf56f3be72bd3e
Add tag names to all unnamed structs in header files. This
allows forward declaration of structs, which is necessary to
reduce header file nesting (to be implemented in a subsequent
commit).
Also change the typedef names across the codebase to use the _t
suffix to be more conformant with the Linux coding style. The
coding style actually prefers us not to use typedefs at all but
this is considered a step too far for Trusted Firmware.
Also change the IO framework structs defintions to use typedef'd
structs to be consistent with the rest of the codebase.
Change-Id: I722b2c86fc0d92e4da3b15e5cab20373dd26786f
Move almost all system include files to a logical sub-directory
under ./include. The only remaining system include directories
not under ./include are specific to the platform. Move the
corresponding source files to match the include directory
structure.
Also remove pm.h as it is no longer used.
Change-Id: Ie5ea6368ec5fad459f3e8a802ad129135527f0b3
AlexeiFedorov
Govindraj Raja
Elyes Haouas
Okash Khawaja
Masahiro Yamada
Varun Wadekar
Raghu Krishnamurthy
Soby Mathew
Julius Werner
Antonio Nino Diaz
Jeenu Viswambharan
Daniel Boulby
Joel Hutton
Etienne Carriere
dp-arm
Andrew Thoelke
Kévin Petit
Achin Gupta
Dan Handley