ARM64 Port Status

oruizdorantes · November 15, 2021, 5:42pm

In my case, the highest is EL3, uboot(2020.04) runs in EL3, but transitions to EL2 during do_bootefi. So the EFI part is running in EL2, and stays until attempting the kernel jump. Another relevant fact is that there is a MMU configuration present, basically mapping the physical regions.

With the same uboot booting Linux(nonefi), there is a EL3>EL2>EL1 transition (existing mmu config gets cleared)

Some questions:

Is everyone expecting to boot the kernel in EL1?
There is in my head as a first approach, to reuse the existing MMU configuration, adding the virtual map for the kernel to be able to jump. Any comments on this?
How valid is the current ARM (armv7) port? Are we supposed to reuse the strategy? (Regarding MMU, I don’t see much low level in the ARM port mainly using kRuntimeServices->SetVirtualAddressMap call)

tqh · November 15, 2021, 8:18pm

You need to setup a mmu map yes, look at x86_64 for what it does before jumping into kernel.
The ARM port is at similar level, some things are ahead and something behind.

You should be able to reuse the kRuntimeServices->SetVirtualAddressMap it is very similar for all platforms.

PulkoMandy · November 15, 2021, 9:31pm

I have no idea. It is probably a “whatever works for you” at the moment.

That’s how it usually works, yes. Bootloader keeps the hardware identity mapped, and adds some mappings in the appropriate places to get the kernel running.

Then kernel takes over, and unmaps the parts handed over from bootloader that are not needed anymore.

Depending on the hardware, the bootloader, and ease of switching from one mmu setup to another, there are many ways to do this. For example, on BIOS, the firmware doesn’t care about MMU and just jumps into our bootloader with MMU disabled, and the bootloader can set up things exactly how the kernel needs them. On EFI, there are constraints on what you can do as long as EFI boot services are running, and so there is some code run late in the boot process, after exiting boot services but before jumping to the kernel. On OpenFirmware, the kernel early boot process will call openfirmware functions, so the handover can be done even later, after starting the kernel and initializing the vm.

The 32bit ARM port is in early stages as well, especially the EFI part of it (previously it was trying to boot from uboot directly). The strategy for all EFI platforms will probably be similar.

andreas_dr · November 29, 2021, 12:14am

Hi, this could be interesting here:

extrowerk · November 29, 2021, 6:34am

435 usd for a motherboard is pretty steep.

andreas_dr · November 29, 2021, 6:45am

Yes. The price is high. Its about 300€ if I got this right, I guess the price is canadian dollars?

BeDifferent-1 · November 29, 2021, 6:50am

$435CA is $341.16US. However, since the page seems to specify USD, I’d say that is a pretty steep price ($435US).

If I had a choice, I’d probably pay about half that amount at most, for a similar product… but that’s just my opinion

Also, it is cool to see the progress being made on the ARM64 and RISC-V ports of Haiku. I hope they eventually have all the capabilities of the x86 and x86_64 versions, although that’s probably still a fairly long way off.

andreas_dr · November 29, 2021, 6:51am

Wow. Yes. Then its super expensive :(((

SWY1985 · November 29, 2021, 8:57am

This little machine is far more interesting for that price: This mini PC has an ARMv8 processor and AMD Radeon RX 550 graphics - Liliputing

Octa core CPU, 16GB ram, and it even has a dedicated AMD GPU. I’d love to one day be able to run Haiku on a similar machine.

oruizdorantes · December 3, 2021, 9:59pm

Hello

As I mentioned I am finding myself with an MMU preconfigured in EL2. That has been useful to understand the MMU itself and how it is being configured. Out of that and digging some ARMv8 documentation:

  [...]
  phys: 0xfbd26000-0xfcf30000, virt: 0xfbd26000-0xfcf30000, size = 0x120a000, type: LoaderData (0x2), attr: 0x8
  phys: 0xfcf30000-0xfcf31000, virt: 0xfcf30000-0xfcf31000, size = 0x1000, type: ReservedMemoryType (0x0), attr: 0x8
  phys: 0xfcf31000-0xfcfc0000, virt: 0xfcf31000-0xfcfc0000, size = 0x8f000, type: LoaderData (0x2), attr: 0x8
  phys: 0xfcfc0000-0xfcfd0000, virt: 0xfcfc0000-0xfcfd0000, size = 0x10000, type: RuntimeServicesCode (0x5), attr: 0x8000000000000008
  phys: 0xfcfd0000-0xfe800000, virt: 0xfcfd0000-0xfe800000, size = 0x1830000, type: LoaderData (0x2), attr: 0x8
  phys: 0xfe800000-0x100000000, virt: 0xfe800000-0x100000000, size = 0x1800000, type: BootServicesData (0x4), attr: 0x8
  phys: 0x880000000-0x8e0000000, virt: 0x880000000-0x8e0000000, size = 0x60000000, type: BootServicesData (0x4), attr: 0x8
Efi loader symbols offset: 0xfbcc0000:
Current Exception Level EL2
MMU Enabled, Translation Table @ 8000c000 Granularity 4KB, bits 40
Kernel entry accessibility W: 0 R: 0
Level 0, @8000c000: TTD 000000008000d003        Page! Next Level:
Level 1, @8000d000: TTD 000000008000e003        Page! Next Level:
Level 2, @8000e000: TTD 0000000000000611        Block   Level 2, @8000e008: TTD 0000000000200611        Block
Level 2, @8000e010: TTD 0000000000400611        Block   Level 2, @8000e018: TTD 0000000000600611        Block
Level 2, @8000e020: TTD 0000000000800611        Block   Level 2, @8000e028: TTD 0000000000a00611        Block
Level 2, @8000e030: TTD 0000000000c00611        Block   Level 2, @8000e038: TTD 0000000000e00611        Block
Level 2, @8000e040: TTD 0000000001000611        Block   Level 2, @8000e048: TTD 0000000001200611        Block
  [...]

Additionally from the memory maps being dumped before, there is some useful information: addreses, exepcion level & MMU translation regime. At last, there is some code developed to handle the table descriptors, useful enough to able to perform a complete tablewalk and dump the whole translation table levels.

Some code base & information dumping, but no actual functional steps forward. Despite that… how about a PR to be in synch?

oruizdorantes · December 21, 2021, 3:18pm

Some updates:

Being more specific about some quotes before (taken from specs):

“The resident UEFI boot-time environment shall use the highest non-secure privilege level available”.

EL2 is the hypervisor level and exists only in non-secure mode. EL3 is the highest priority level and exists only in secure mode.

So in summary if available we will be (max) in EL2. Remarking that finding ourselves in EL1 is still possible.

EL2 and EL3 have a TTBR0, but no TTBR1. This means that is either EL2 or EL3 is using AArch64, they can only use virtual addresses in the range 0x0 to 0x0000FFFF_FFFFFFFF.

Summary: EL2’s MMU doesn’t have EL1’s potential (This quote is partially true, FEAT_VHE would give an EL2 MMU the capabilities of EL1, but that is not present in a CortexA53 at least)

This fact and, the obvious, (EL2 is for Hypervisors, and eventually would not want to conflict with them) Force us to make a EL2 >> EL1 transition anyway (unless if we find ourselves in EL1)

On the Practical side, I believe the code for mapping the mmu ranges is working. Still need work on the descriptor attributes & caches on the tables…

[...]
map 0xffffffff80000000 --> 0xf9aa5000, len=0x20a000, flags=0x7
Level 0, Processing desc f9694000 indexing f9694ff8
Level 1, Processing desc f968e000 indexing f968eff0
Level 2, Processing desc f968d000 indexing f968d000
map 0xffff000002000000 --> 0xf9caf000, len=0x10000, flags=0x7
Level 0, Processing desc f9694000 indexing f9694000
Level 1, Processing desc f9693000 indexing f9693000
Level 2, Processing desc f9692000 indexing f9692080
Level 3, Processing desc f968f000 indexing f968f000
map 0x9000000 --> 0x9000000, len=0x1000, flags=0x1
Range already covered in current MMU
map 0x0 --> 0x0, len=0x18000, flags=0x7
Range already covered in current MMU

There will be still some register work to do, and if I am not missing anything next the kernel jump… wondering if this could done together at the EL2>EL1 switch…

tqh · December 21, 2021, 5:30pm

Nice, keep it up! I don’t have much time at the moment, and most time would be on build side of things anyway.

Andrea · January 6, 2022, 6:47pm

any news?

oruizdorantes · January 10, 2022, 10:16pm

eeehmmm no

Little steps forward, attributes, there is a EL1 transition… but I guess the tables are not right yet…

oruizdorantes · January 18, 2022, 6:22pm

Now this mapping is the expected.

map 0xffffffff80000000 --> 0xf9aa5000, len=0x20a000

Señores:

We have a kernel jump

After all the mapping the arch_enter_kernel is reached, jumps to virtual space, and even takes a while to crash again…

There is an interesting instruction called AT which lets you test if an address is mmu-accesible or not. This has avoided many unnecessary crashes until the MMU was corrected. (maybe this is typical for processors but I come from mmu-less microcontrollers, just started in armv8 architecture as posting here)

The work additionally to the whole MMU handling, takes the logics with EFI reused from the ARM port and there is some reuse of Jarek’s work (EL2>EL1 transition)

There are still things left behind and… still a long way to go.

Time to stop, cleanup (I reused some cache handling GPLed code, any indication for alternative code will be appreciated) and make it to the tree.

tqh · January 30, 2022, 6:25pm

Do not expect updates from me in this topic from now on. I think the ARM64 port is in good hands, I lack time and ARM64 skills to continue this work.
I’ll be following with interest, helping out if needed, but I want to start working on other (smaller) things.

oruizdorantes · January 30, 2022, 8:01pm

Without your feedback, I would not have been enough motivated,
Without you bringing the port to the point It was, I would not have been capable to continue, as I lack other set of skills, such as ELF Management, EFI, and a ready to go Build system…

Thanks & Keep around!

X512 · January 30, 2022, 8:47pm

You can ask me too, an author of RISC-V port.

PulkoMandy · January 31, 2022, 7:42am

Nice to see there is great teamwork happening

oruizdorantes · March 8, 2022, 3:30pm

I have taken as next step enabling the present UART in my board. All required information is located is in the DTB loaded by uboot/efi. Same one is used as if there was a Linux kernel.

The information could be retrieved enabling all the Device Tree handling exisiting code without major issues.

The UART present in my SoC is called LINFlex, is tipically present in NxP(freescale) and STMicroelectronics devices. A new driver has been required, but the UART itself is already initalized by uboot.

Therefore in a first approach by only implementing the putchar & getchar methods, we see firsts prints comming from the Kernel iself:

LINFlex UART is also seen in ARM(armv7 32 bits) even it can be seen in PPC parts, this is not yet a problem.

But it is already a problem the PL011 present in QEMU, which would also apply for the ARM64 port, I have been in the situation of duplicating PL011 code or referencing ARM files from ARM64 port in the jamfiles, this would need a discussion…