Userspace networking on a machine with an OS?

[0xffffa]

Hey first I want to start of by saying amazing libraries, it is always a pleasure to be able to encounter hard working people who constantly give back and add to the golang ecosystem I love and enjoy so much!

With that being said, I have a few questions and problems I have been running into in related to the project as my setup is slightly unique.

I am using TinyGo to compile a golang project which will be run on a regular machine with an os (rarely ever embedded or bare metal systems). The reason I'm doing this as I need to make many requests on systems with extreme memory constraints (64MB - 128MB sometimes maybe even less), and there is too much overhead in the current tinygo networking stack. My original plan was to use fasthttp however that is not supported by tinygo.

With that being said, how can I implement the os source routing into lneto, taking advantage of lneto's small memory footprint and processing speed while letting the os do all the egress backend work. Is this possible or is the library not suitable for this specific task. Thank you!

[soypat]

Heya @0xffffa! The answer is "Most certainly yes!". I'm currently running lneto's internet stack with a HTTP server at https://lneto.whittileaks.com which you can check out on a very constrained system (264kB RAM, 2MB flash). The source code is over at https://github.com/soypat/lan8720/blob/main/examples/pico-http-server/main.go. Note it is more verbose since I had to make it fully heapless, no heap allocations after initialization which requires some careful management of goroutines and HTTP buffers. Below is the hardware setup.

Since you are using an OS you could make use of the httpraw package and the OS's TCP functionality. httpraw offers many ways to process incoming requests depending on your system's constraints. Specifically take a look at how the following httpraw.Header methods work:

• func (h *Header) ParseBytes(asResponse bool, b []byte) error parses a header from a buffer. Performs a copy so is more expensive
• You can instead of using ParseBytes call Reset with the buffer and then TryParse to omit copy and share the memory
• All methods that start with Read* add data to Header's underlying buffer to then use TryParse instead of ParseBytes
• See func (h *Header) Body() ([]byte, error) for extra body bytes. This is surplus data read beyond final \r\n\r\n
• func (h *Header) Reset(buf []byte) after finished using

Let me know how that goes! Any question regarding use or bugs please feel free to file an issue!

[0xffffa]

I may be missing something with the httpraw package, but is there a way to set the host header? Im having trouble doing so without post byte array modification

[soypat]

You'd use the Set and SetBytes methods. Some header setting is provided free of charge as static methods. Some headers were deemed not as special when writing httpraw. If you are interested in having more methods added it may be worth a discussion or issue.

Do note I detected bugs when scanning the httpraw code! #159

[0xffffa]

Great and no problem im mainly looking at it for a small project. Youve been a great help, the last issue I have is when attempting to parse an http response gotten from a socket connect it just returns "missing protocol, HTTP/0.9 unsupported" regardless of how i read the bytes whether it be directly or not

code here: (sorry on phone)

buf := make([]byte, 1024)
syscall.Read(fd, buf)

hdr.ReadFromBytes(buf) // passes with no errors

hdr.TryParse(true) // returns false

println(hdr.String()) // the problem prints missing protocol

[soypat]

Could you print the buf on receive and show me what the header looks like??

println(string(buf))
``

[soypat]

Also I'd be interested in you helping out creating an example of using the httpraw library that would make sense to you. We don't have too many lneto-on-OS users and I'd really like to understand the main barriers to using it :)

[soypat]

I've added an example of how I'd imagine httpraw would best be used with minimal operating system functionality. I noticed some functions were lacking on httpraw's side and also noticed a few bugs whose fixes have been merged since. As for now here's my vision of how it would be used.

https://github.com/soypat/lneto/blob/main/examples/http-linux/main-httplinux.go

You may notice some tacky patterns beyond the HTML design like buffer reuse. I like to provide examples showing how to avoid all heap allocations. Users can then choose to simplify if they don't care about that style.