When acquiring or releasing the power domain locks for a given CPU the
parent nodes are looked up by walking the up the PD tree list on both the
acquire and release path, only one set of lookups is needed. Fetch the
parent nodes first and pass this list into both the acquire and release
functions to avoid the double lookup.
This also allows us to not have to do this lookup after coherency has
been exited during the core power down sequence. The shared struct
psci_cpu_pd_nodes is not placed in coherent memory like is done
for psci_non_cpu_pd_nodes and doing so would negatively affect
performance. With this patch we remove the need to have it in coherent
memory by moving the access out of psci_release_pwr_domain_locks().
Signed-off-by: Andrew F. Davis <afd@ti.com>
Change-Id: I7b9cfa9d31148dea0f5e21091c8b45ef7fe4c4ab
Enforce full include path for includes. Deprecate old paths.
The following folders inside include/lib have been left unchanged:
- include/lib/cpus/${ARCH}
- include/lib/el3_runtime/${ARCH}
The reason for this change is that having a global namespace for
includes isn't a good idea. It defeats one of the advantages of having
folders and it introduces problems that are sometimes subtle (because
you may not know the header you are actually including if there are two
of them).
For example, this patch had to be created because two headers were
called the same way: e0ea0928d5 ("Fix gpio includes of mt8173 platform
to avoid collision."). More recently, this patch has had similar
problems: 46f9b2c3a2 ("drivers: add tzc380 support").
This problem was introduced in commit 4ecca33988 ("Move include and
source files to logical locations"). At that time, there weren't too
many headers so it wasn't a real issue. However, time has shown that
this creates problems.
Platforms that want to preserve the way they include headers may add the
removed paths to PLAT_INCLUDES, but this is discouraged.
Change-Id: I39dc53ed98f9e297a5966e723d1936d6ccf2fc8f
Signed-off-by: Antonio Nino Diaz <antonio.ninodiaz@arm.com>
Initial Spectre Variant 1 mitigations (CVE-2017-5753).
A potential speculative data leak was found in PSCI code, this depends
on a non-robust implementation of the `plat_get_core_pos_by_mpidr()`
function. This is considered very low-risk. This patch adds a macro to
mitigate this. Note not all code paths could be analyzed with current
tools.
Add a macro which makes a variable 'speculation safe', using the
__builtin_speculation_safe_value function of GCC and llvm. This will be
available in GCC 9, and is planned for llvm, but is not currently in
mainline GCC or llvm. In order to implement this mitigation the compiler
must support this builtin. Support is indicated by the
__HAVE_SPECULATION_SAFE_VALUE flag.
The -mtrack-speculation option maintains a 'tracker' register, which
determines if the processor is in false speculation at any point. This
adds instructions and increases code size, but avoids the performance
impact of a hard barrier.
Without the -mtrack-speculation option, __builtin_speculation_safe_value
expands to a
ISB
DSB SY
sequence after a conditional branch, before the
speculation safe variable is used. With -mtrack-speculation a
CSEL tracker, tracker, XZR, [cond];
AND safeval,tracker;
CSDB
sequence is added instead, clearing the vulnerable variable by
AND'ing it with the tracker register, which is zero during speculative
execution. [cond] are the status flags which will only be true during
speculative execution. For more information on
__builtin_speculation_safe_value and the -mtrack-speculation option see
https://developer.arm.com/support/arm-security-updates/speculative-processor-vulnerability/compiler-support-for-mitigations
The -mtracking option was not added, as the performance impact of the
mitigation is low, and there is only one occurence.
Change-Id: Ic9e66d1f4a5155e42e3e4055594974c230bfba3c
Signed-off-by: Joel Hutton <Joel.Hutton@Arm.com>
Some platforms can only resume from system suspend from the boot
CPU, hence they should only enter that state from that same core.
The following commit presents an interface that allows the platform to
reject system suspend entry near its very last stage (last CPU).
When a platform enables its caches before it accesses the
psci_non_cpu_pd_nodes structure then explicit cache maintenance
is not needed.
Signed-off-by: Andrew F. Davis <afd@ti.com>
The MMU is not disabled in this path, update the comment to
reflect this. Also clarify that both paths call prepare_cpu_pwr_dwn(),
but the second path does stack cache maintenance.
Signed-off-by: Andrew F. Davis <afd@ti.com>
Mark the initialization functions in BL31, such as context management,
EHF, RAS and PSCI as __init so that they can be reclaimed by the
platform when no longer needed
Change-Id: I7446aeee3dde8950b0f410cb766b7a2312c20130
Signed-off-by: Daniel Boulby <daniel.boulby@arm.com>
During cold boot, the initial translation tables are created with data
caches disabled, so all modifications go to memory directly. After the
MMU is enabled and data cache is enabled, any modification to the tables
goes to data cache, and eventually may get flushed to memory.
If CPU0 modifies the tables while CPU1 is off, CPU0 will have the
modified tables in its data cache. When CPU1 is powered on, the MMU is
enabled, then it enables coherency, and then it enables the data cache.
Until this is done, CPU1 isn't in coherency, and the translation tables
it sees can be outdated if CPU0 still has some modified entries in its
data cache.
This can be a problem in some cases. For example, the warm boot code
uses only the tables mapped during cold boot, which don't normally
change. However, if they are modified (and a RO page is made RW, or a XN
page is made executable) the CPU will see the old attributes and crash
when it tries to access it.
This doesn't happen in systems with HW_ASSISTED_COHERENCY or
WARMBOOT_ENABLE_DCACHE_EARLY. In these systems, the data cache is
enabled at the same time as the MMU. As soon as this happens, the CPU is
in coherency.
There was an attempt of a fix in psci_helpers.S, but it didn't solve the
problem. That code has been deleted. The code was introduced in commit
<264410306381> ("Invalidate TLB entries during warm boot").
Now, during a map or unmap operation, the memory associated to each
modified table is flushed. Traversing a table will also flush it's
memory, as there is no way to tell in the current implementation if the
table that has been traversed has also been modified.
Change-Id: I4b520bca27502f1018878061bc5fb82af740bb92
Signed-off-by: Antonio Nino Diaz <antonio.ninodiaz@arm.com>
If either USE_COHERENT_MEM or HW_ASSISTED_COHERENCY being true
should cause us to not enter the ifdef block, then the logic
is not correct here. Posibly bad use of De Morgan's law?
Fix this.
Signed-off-by: Andrew F. Davis <afd@ti.com>
Use a _ prefix for Macro arguments to prevent that argument from
hiding variables of the same name in the outer scope
Rule 5.3: An identifier declared in an inner scope shall not
hide an identifier declared in an outer scope
Fixed For:
make LOG_LEVEL=50 PLAT=fvp
Change-Id: I67b6b05cbad4aeca65ce52981b4679b340604708
Signed-off-by: Daniel Boulby <daniel.boulby@arm.com>
Initializes each element of the last_cpu_in_non_cpu_pd array in PSCI
stat implementation to -1, the reset value. This satisfies MISRA rule
9.3.
Previously, only the first element of the array was initialized to -1.
Change-Id: I666c71e6c073710c67c6d24c07a219b1feb5b773
Signed-off-by: Jonathan Wright <jonathan.wright@arm.com>
When the source code says 'SMCC' it is talking about the SMC Calling
Convention. The correct acronym is SMCCC. This affects a few definitions
and file names.
Some files have been renamed (smcc.h, smcc_helpers.h and smcc_macros.S)
but the old files have been kept for compatibility, they include the
new ones with an ERROR_DEPRECATED guard.
Change-Id: I78f94052a502436fdd97ca32c0fe86bd58173f2f
Signed-off-by: Antonio Nino Diaz <antonio.ninodiaz@arm.com>
Rule 8.3: All declarations of an object or function shall
use the same names and type qualifiers.
Change-Id: Iff384187c74a598a4e73f350a1893b60e9d16cec
Signed-off-by: Roberto Vargas <roberto.vargas@arm.com>
During the warm boot sequence:
1. The MMU is enabled with the data cache disabled. The MMU table walker
is set up to access the translation tables as in cacheable memory,
but its accesses are non-cacheable because SCTLR_EL3.C controls them
as well.
2. The interconnect is set up and the CPU enters coherency with the
rest of the system.
3. The data cache is enabled.
If the support for dynamic translation tables is enabled and another CPU
makes changes to a region, the changes may only be present in the data
cache, not in RAM. The CPU that is booting isn't in coherency with the
rest of the system, so the table walker of that CPU isn't either. This
means that it may read old entries from RAM and it may have invalid TLB
entries corresponding to the dynamic mappings.
This is not a problem for the boot code because the mapping is 1:1 and
the regions are static. However, the code that runs after the boot
sequence may need to access the dynamically mapped regions.
This patch invalidates all TLBs during warm boot when the dynamic
translation tables support is enabled to prevent this problem.
Change-Id: I80264802dc0aa1cb3edd77d0b66b91db6961af3d
Signed-off-by: Antonio Nino Diaz <antonio.ninodiaz@arm.com>
On some platforms it may be necessary to discover the SMCCC version
via a PSCI features call.
Change-Id: I95281ac2263ca9aefda1809eb03464fbdb8ac24d
Signed-off-by: Dimitris Papastamos <dimitris.papastamos@arm.com>
The suspend hook is published at the start of a CPU powerdown
operation. The resume hook is published at the end of a CPU powerup
operation.
Change-Id: I50c05e2dde0d33834095ac41b4fcea4c161bb434
Signed-off-by: Dimitris Papastamos <dimitris.papastamos@arm.com>
There is an edge case where the cache maintaince done in
psci_do_cpu_off may not seen by some cores. This case is handled in
psci_cpu_on_start but it hasn't handled in psci_affinity_info.
Change-Id: I4d64f3d1ca9528e364aea8d04e2d254f201e1702
Signed-off-by: Roberto Vargas <roberto.vargas@arm.com>
If ARMv7 based platform does not set ARM_CORTEX_Ax=yes, platform
shall define ARMV7_SUPPORTS_GENERIC_TIMER to enable generic timer
support.
Signed-off-by: Etienne Carriere <etienne.carriere@linaro.org>
This patch includes various fixes for PSCI STAT functionality
relating to timestamp collection:
1. The PSCI stat accounting for retention states for higher level
power domains were done outside the locks which could lead to
spurious values in some race conditions. This is moved inside
the locks. Also, the call to start the stat accounting was redundant
which is now removed.
2. The timestamp wrap-around case when calculating residency did
not cater for AArch32. This is now fixed.
3. In the warm boot path, `plat_psci_stat_accounting_stop()` was
getting invoked prior to population of target power states. This
is now corrected.
Change-Id: I851526455304fb74ff0a724f4d5318cd89e19589
Signed-off-by: Soby Mathew <soby.mathew@arm.com>
This allows other EL3 components to subscribe to CPU on events.
Update Firmware Design guide to list psci_cpu_on_finish as an available
event.
Change-Id: Ida774afe0f9cdce4021933fcc33a9527ba7aaae2
Signed-off-by: Jeenu Viswambharan <jeenu.viswambharan@arm.com>
With this patch the PSCI_FEATURES API correctly reports availability
of the PSCI_MEM_PROTECT_CHECK API - PSCI_MEM_CHK_RANGE_AARCH64 is
added to the PSCI capabilities mask, PSCI_CAP_64BIT_MASK
Change-Id: Ic90ee804deaadf0f948dc2d46ac5fe4121ef77ae
Signed-off-by: Roberto Vargas <roberto.vargas@arm.com>
This patch implements PSCI_SYSTEM_RESET2 API as defined in PSCI
v1.1 specification. The specification allows architectural and
vendor-specific resets via this API. In the current specification,
there is only one architectural reset, the warm reset. This reset is
intended to provide a fast reboot path that guarantees not to reset
system main memory.
Change-Id: I057bb81a60cd0fe56465dbb5791d8e1cca025bd3
Signed-off-by: Roberto Vargas <roberto.vargas@arm.com>
This patch adds the generic code that links the psci smc handler
with the platform function that implements the mem_protect and
mem_check_range functionalities. These functions are optional
APIs added in PSCI v1.1 (ARM DEN022D).
Change-Id: I3bac1307a5ce2c7a196ace76db8317e8d8c8bb3f
Signed-off-by: Roberto Vargas <roberto.vargas@arm.com>
psci_set_power_off_state only initializes a local variable, so there
isn't any reason why it should be done while the lock is held.
Change-Id: I1c62f4cd5d860d102532e5a5350152180d41d127
Signed-off-by: Roberto Vargas <roberto.vargas@arm.com>
Previously commit 555ebb34db8f3424c1b394df2f10ecf9c1f70901 attmpted to fix this
GCC issue:
services/std_svc/psci/psci_common.c: In function 'psci_do_state_coordination':
services/std_svc/psci/psci_common.c:220:27: error: array subscript is above
array bounds [-Werror=array-bounds]
psci_req_local_pwr_states[pwrlvl - 1][cpu_idx] = req_pwr_state;
This fix doesn't work as asserts aren't built in non-debug build flows.
Let's use GCCs #pragma option (documented here:
https://gcc.gnu.org/onlinedocs/gcc/Diagnostic-Pragmas.html) to avoid
this false positive instead.
Signed-off-by: Alistair Francis <alistair.francis@xilinx.com>
This patch adds an early suspend handler, that executes with
SMP and data cache enabled. This handler allows platforms to
perform any early actions during the CPU suspend entry sequence.
This handler is optional and platforms can choose to implement it
depending on their needs. The `pwr_domain_suspend` handler still
exists and platforms can keep on using it without any side effects.
Signed-off-by: Varun Wadekar <vwadekar@nvidia.com>
There is a theoretical edge case during CPU_ON where the cache
may contain stale data for the target CPU data - this can occur
under the following conditions:
- the target CPU is in another cluster from the current
- the target CPU was the last CPU to shutdown on its cluster
- the cluster was removed from coherency as part of the CPU shutdown
In this case the cache maintenace that was performed as part of the
target CPUs shutdown was not seen by the current CPU's cluster. And
so the cache may contain stale data for the target CPU.
This patch adds a cache maintenance operation (flush) for the
cache-line containing the target CPU data - this ensures that the
target CPU data is read from main memory.
Change-Id: If8cfd42639b03174f60669429b7f7a757027d0fb
Signed-off-by: David Cunado <david.cunado@arm.com>
This fix modifies the order of system includes to meet the ARM TF coding
standard. There are some exceptions in order to retain header groupings,
minimise changes to imported headers, and where there are headers within
the #if and #ifndef statements.
Change-Id: I65085a142ba6a83792b26efb47df1329153f1624
Signed-off-by: Isla Mitchell <isla.mitchell@arm.com>
Call svc_suspend_finish if registered.
psci_get_stat() is static to psci_stat.c
Fix types used in comparison.
Fix coding style (empty line between variable definition and instructions
block).
Signed-off-by: Etienne Carriere <etienne.carriere@st.com>
This patch enables the 'sign-compare' flag, to enable warning/errors
for comparisons between signed/unsigned variables. The warning has
been enabled for all the Tegra platforms, to start with.
Signed-off-by: Varun Wadekar <vwadekar@nvidia.com>
In AArch64, privileged exception levels control the execution state
(a.k.a. register width) of the immediate lower Exception Level; i.e.
whether the lower exception level executes in AArch64 or AArch32 state.
For an exception level to have its execution state changed at run time,
it must request the change by raising a synchronous exception to the
higher exception level.
This patch implements and adds such a provision to the ARM SiP service,
by which an immediate lower exception level can request to switch its
execution state. The execution state is switched if the request is:
- raised from non-secure world;
- raised on the primary CPU, before any secondaries are brought online
with CPU_ON PSCI call;
- raised from an exception level immediately below EL3: EL2, if
implemented; otherwise NS EL1.
If successful, the SMC doesn't return to the caller, but to the entry
point supplied with the call. Otherwise, the caller will observe the SMC
returning with STATE_SW_E_DENIED code. If ARM Trusted Firmware is built
for AArch32, the feature is not supported, and the call will always
fail.
For the ARM SiP service:
- Add SMC function IDs for both AArch32 and AArch64;
- Increment the SiP service minor version to 2;
- Adjust the number of supported SiP service calls.
Add documentation for ARM SiP service.
FixesARM-software/tf-issues#436
Change-Id: I4347f2d6232e69fbfbe333b340fcd0caed0a4cea
Signed-off-by: Jeenu Viswambharan <jeenu.viswambharan@arm.com>
To make software license auditing simpler, use SPDX[0] license
identifiers instead of duplicating the license text in every file.
NOTE: Files that have been imported by FreeBSD have not been modified.
[0]: https://spdx.org/
Change-Id: I80a00e1f641b8cc075ca5a95b10607ed9ed8761a
Signed-off-by: dp-arm <dimitris.papastamos@arm.com>
This patch introduces a build option to enable D-cache early on the CPU
after warm boot. This is applicable for platforms which do not require
interconnect programming to enable cache coherency (eg: single cluster
platforms). If this option is enabled, then warm boot path enables
D-caches immediately after enabling MMU.
FixesARM-Software/tf-issues#456
Change-Id: I44c8787d116d7217837ced3bcf0b1d3441c8d80e
Signed-off-by: Soby Mathew <soby.mathew@arm.com>
Call console_flush() before execution either terminates or leaves an
exception level.
Fixes: ARM-software/tf-issues#123
Change-Id: I64eeb92effb039f76937ce89f877b68e355588e3
Signed-off-by: Antonio Nino Diaz <antonio.ninodiaz@arm.com>
The current PSCI implementation can apply certain optimizations upon the
assumption that all PSCI participants are cache-coherent.
- Skip performing cache maintenance during power-up.
- Skip performing cache maintenance during power-down:
At present, on the power-down path, CPU driver disables caches and
MMU, and performs cache maintenance in preparation for powering down
the CPU. This means that PSCI must perform additional cache
maintenance on the extant stack for correct functioning.
If all participating CPUs are cache-coherent, CPU driver would
neither disable MMU nor perform cache maintenance. The CPU being
powered down, therefore, remain cache-coherent throughout all PSCI
call paths. This in turn means that PSCI cache maintenance
operations are not required during power down.
- Choose spin locks instead of bakery locks:
The current PSCI implementation must synchronize both cache-coherent
and non-cache-coherent participants. Mutual exclusion primitives are
not guaranteed to function on non-coherent memory. For this reason,
the current PSCI implementation had to resort to bakery locks.
If all participants are cache-coherent, the implementation can
enable MMU and data caches early, and substitute bakery locks for
spin locks. Spin locks make use of architectural mutual exclusion
primitives, and are lighter and faster.
The optimizations are applied when HW_ASSISTED_COHERENCY build option is
enabled, as it's expected that all PSCI participants are cache-coherent
in those systems.
Change-Id: Iac51c3ed318ea7e2120f6b6a46fd2db2eae46ede
Signed-off-by: Jeenu Viswambharan <jeenu.viswambharan@arm.com>
The PSCI implementation performs cache maintenance operations on its
data structures to ensure their visibility to both cache-coherent and
non-cache-coherent participants. These cache maintenance operations
can be skipped if all PSCI participants are cache-coherent. When
HW_ASSISTED_COHERENCY build option is enabled, we assume PSCI
participants are cache-coherent.
For usage abstraction, this patch introduces wrappers for PSCI cache
maintenance and barrier operations used for state coordination: they are
effectively NOPs when HW_ASSISTED_COHERENCY is enabled, but are
applied otherwise.
Also refactor local state usage and associated cache operations to make
it clearer.
Change-Id: I77f17a90cba41085b7188c1345fe5731c99fad87
Signed-off-by: Jeenu Viswambharan <jeenu.viswambharan@arm.com>
Perform stat accounting for retention/standby states also when
requested at multiple power levels.
Change-Id: I2c495ea7cdff8619bde323fb641cd84408eb5762
Signed-off-by: dp-arm <dimitris.papastamos@arm.com>
Andrew F. Davis
Antonio Nino Diaz
Joel Hutton
ldts
Daniel Boulby
Roberto Vargas
Jonathan Wright
Dimitris Papastamos
Etienne Carriere
Soby Mathew
Jeenu Viswambharan
Alistair Francis
Varun Wadekar
David Cunado
Isla Mitchell
Etienne Carriere