Skip to content

there must be a bug. I get a lot of RSTs when browsing in a netns with a TUN driven by it #6

Closed
Closed
there must be a bug. I get a lot of RSTs when browsing in a netns with a TUN driven by it#6
Assignees
SajjadPourali
Labels
bugSomething isn't working

Description

@ple1n
ple1n
opened on Dec 12, 2023
 nsproxy[214777]: 2023-12-12T11:32:44.167596Z  INFO ipstack: NetworkPacket { ip: Version4(Ipv4Header { ihl: 5, differentiated_services_code_point: 0, explicit_congestion_notification: 0, payload_len: 477, identification: 0, dont_fragment: true, more_fragments: false, fragments_offset: 0, time_to_live: 64, protocol: 6, header_checksum: 0, source: [198, 18, 0, 0], destination: [100, 64, 0, 2], options: [] }, Ipv4Extensions { auth: None }), transport: Tcp(TcpHeader { source_port: 80, destination_port: 33816, sequence_number: 1251, acknowledgment_number: 1444113908, data_offset: 5, ns: false, fin: false, syn: false, rst: false, psh: true, ack: true, urg: false, ece: false, cwr: false, window_size: 16384, checksum: 27942, urgent_pointer: 0, options: [] }) }
 nsproxy[214777]: 2023-12-12T11:32:44.168245Z  INFO ipstack: NetworkPacket { ip: Version4(Ipv4Header { ihl: 5, differentiated_services_code_point: 0, explicit_congestion_notification: 0, payload_len: 20, identification: 0, dont_fragment: true, more_fragments: false, fragments_offset: 0, time_to_live: 64, protocol: 6, header_checksum: 0, source: [198, 18, 0, 0], destination: [100, 64, 0, 2], options: [] }, Ipv4Extensions { auth: None }), transport: Tcp(TcpHeader { source_port: 80, destination_port: 33816, sequence_number: 1708, acknowledgment_number: 1444113909, data_offset: 5, ns: false, fin: true, syn: false, rst: false, psh: false, ack: true, urg: false, ece: false, cwr: false, window_size: 16384, checksum: 63073, urgent_pointer: 0, options: [] }) }
 nsproxy[214777]: 2023-12-12T11:32:49.169809Z TRACE ipstack::stream::tcp: timeout reached for 198.18.0.0:80
 nsproxy[214777]: 2023-12-12T11:32:49.170011Z  INFO tun2socks5: Ending #1 TCP 100.64.0.2:33816 -> example.com.:80 with Err(Kind(TimedOut))
 nsproxy[214777]: 2023-12-12T11:32:49.170298Z TRACE tun2socks5: Session count 1

I'm trying to solve it. A whole TCP stack written by hand. Wcgw.

Activity

  1. ple1n commented on Dec 12, 2023

    @ple1n
    ple1n
    on Dec 12, 2023
    Author

    image

    check_pkt_type seems buggy. The last packet should be ACKing the FIN but it's invalidated

  2. ple1n commented on Dec 12, 2023

    @ple1n
    ple1n
    on Dec 12, 2023
    Author

    I did a workaround. ple1n/ipstack@2dc5516

  3. SajjadPourali
    self-assigned this
    on Dec 12, 2023

  4. SajjadPourali commented on Dec 12, 2023

    @SajjadPourali
    SajjadPourali
    on Dec 12, 2023
    Collaborator

    image

    check_pkt_type seems buggy. The last packet should be ACKing the FIN but it's invalidated

    I hope the problem is limited only to check_pkt_type because it would be easy to fix.

    Could you please provide a pcap file and mention the OS that you use?

  5. SajjadPourali
    added
    bugSomething isn't working
    on Dec 13, 2023

  6. ple1n commented on Dec 13, 2023

    @ple1n
    ple1n
    on Dec 13, 2023
    Author

    heres the cap of curl example.com on linux 6.6.5-arch1-1 #1 SMP PREEMPT_DYNAMIC. https://github.com/planetoryd/misc/blob/master/cap_ipstack_rst.pcapng with https://github.com/ssrlive/ipstack/commit/df0765ab7636f7e9d2846b779bf412e15a205faa

    with dbg!s https://github.com/planetoryd/misc/blob/master/ipstack_logs.md

    the workarounded version https://github.com/planetoryd/misc/blob/master/ipstack_workaround.pcapng has RST+ACKs, which should be RSTing the connection and immediate restablishment (i guess) . on ple1n/ipstack@2dc5516

    If you see large captures it's me using it as some kind of daily driver. therefore just ask if you want any data, or testing. and I use geph.io 's proxy as upstream, which is a tor-bridge like proxy service (better than tor bridge in fact)

    It looks quite terrifying to have that much unverified logic. perhaps you can try https://github.com/verus-lang/verus but idk how long it will take to become usable. some nice formal verification (unlikely)

  7. SajjadPourali commented on Dec 13, 2023

    @SajjadPourali
    SajjadPourali
    on Dec 13, 2023
    Collaborator

    heres the cap of curl example.com on linux 6.6.5-arch1-1 #1 SMP PREEMPT_DYNAMIC. https://github.com/planetoryd/misc/blob/master/cap_ipstack_rst.pcapng with ssrlive@df0765a

    with dbg!s https://github.com/planetoryd/misc/blob/master/ipstack_logs.md

    the workarounded version https://github.com/planetoryd/misc/blob/master/ipstack_workaround.pcapng has RST+ACKs, which should be RSTing the connection and immediate restablishment (i guess) . on planetoryd@2dc5516

    If you see large captures it's me using it as some kind of daily driver. therefore just ask if you want any data, or testing. and I use geph.io 's proxy as upstream, which is a tor-bridge like proxy service (better than tor bridge in fact)

    It looks quite terrifying to have that much unverified logic. perhaps you can try https://github.com/verus-lang/verus but idk how long it will take to become usable. some nice formal verification (unlikely)

    Fortunately, the logic is almost correct. There is only one minor issue that should not affect the communications, but it is indeed a bug! The IP stack deallocates the TCP stream after receiving the last FIN/ACK, so the final ACK goes unrecognized, which results in a RST being returned. In the correct implementation, it should handle the last ACK as well. However, in both cases—whether it returns an RST (the faulty behavior) or accepts the ACK without returning an RST — the connection will be closed.

    Thanks for the report, I'll fix it soon. Did you see any other unexpected rst except the last ack for the first one?

    I still did not checked your modifications on ple1n/ipstack@2dc5516 .

  8. SajjadPourali commented on Dec 13, 2023

    @SajjadPourali
    SajjadPourali
    on Dec 13, 2023
    Collaborator

    @planetoryd: Could you please verify that if #7 fixes the issue?

  9. ssrlive commented on Dec 13, 2023

    @ssrlive
    ssrlive
    on Dec 13, 2023
    Collaborator

    It's not fixed. @SajjadPourali

    You can test ipstack with tun2socks5 under Linux. steps like this

    sudo apt install iperf3 dante-server
sudo systemctl stop danted

git clone https://github.com/ssrlive/tun2socks5.git
cd tun2socks5

cargo b -r

# start testing
sudo scripts/iperf3.sh

# clear up
sudo sh -c "pkill tun2socks5; pkill iperf3; pkill danted; ip link del tun0; ip netns del test"

  10. ple1n commented on Dec 13, 2023

    @ple1n
    ple1n
    on Dec 13, 2023
    Author

    Yeah, the following RST got removed, when I did the curl again.

    I open my browser, mozilla connects something. It constantly times out, RSTs the connections and reconnect immediately.

    https://github.com/planetoryd/misc/blob/master/timeout.pcapng

  11. ple1n commented on Dec 13, 2023

    @ple1n
    ple1n
    on Dec 13, 2023
    Author
    2023-12-13T15:04:48.300632Z  INFO tun2socks5::dns: Allocate VirtDNS Name example.com.
2023-12-13T15:04:48.301102Z  INFO tun2socks5::dns: Allocate VirtDNS Name example.com.
2023-12-13T15:04:48.302699Z  INFO tun2socks5: Beginning #1 TCP 100.64.0.2:49738 -> example.com.:80
2023-12-13T15:06:18.691060Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:18.691528Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:18.695496Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:18.695741Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:18.697228Z  INFO tun2socks5: Beginning #4 TCP 100.64.0.2:39226 -> push.services.mozilla.com.:443
2023-12-13T15:06:19.448777Z  INFO tun2socks5::dns: Allocate VirtDNS Name r3.o.lencr.org.
2023-12-13T15:06:19.449152Z  INFO tun2socks5::dns: Allocate VirtDNS Name r3.o.lencr.org.
2023-12-13T15:06:19.450398Z  INFO tun2socks5: Beginning #6 TCP 100.64.0.2:48484 -> r3.o.lencr.org.:80
2023-12-13T15:06:20.047536Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:20.047886Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:20.049742Z  INFO tun2socks5: Beginning #8 TCP 100.64.0.2:39238 -> push.services.mozilla.com.:443
2023-12-13T15:06:20.971163Z  INFO tun2socks5: Beginning #9 UDP 100.64.0.2:49230 -> push.services.mozilla.com.:443
2023-12-13T15:06:21.274665Z ERROR tun2socks5: #9 UDP 100.64.0.2:49230 -> push.services.mozilla.com.:443 error "Invalid argument (os error 22)"
2023-12-13T15:06:24.841364Z  INFO tun2socks5: Ending #6 TCP 100.64.0.2:48484 -> r3.o.lencr.org.:80 with Err(Kind(TimedOut))
2023-12-13T15:06:25.047877Z  INFO tun2socks5: Ending #4 TCP 100.64.0.2:39226 -> push.services.mozilla.com.:443 with Err(Kind(TimedOut))
2023-12-13T15:06:26.389020Z  INFO tun2socks5: Ending #8 TCP 100.64.0.2:39238 -> push.services.mozilla.com.:443 with Err(Kind(TimedOut))
2023-12-13T15:06:31.394497Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:31.397211Z  INFO tun2socks5: Beginning #11 TCP 100.64.0.2:60064 -> push.services.mozilla.com.:443
2023-12-13T15:06:31.646661Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:31.648910Z  INFO tun2socks5: Beginning #13 TCP 100.64.0.2:60068 -> push.services.mozilla.com.:443
2023-12-13T15:06:32.132862Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:32.133311Z  INFO tun2socks5::dns: Allocate VirtDNS Name push.services.mozilla.com.
2023-12-13T15:06:32.135715Z  INFO tun2socks5: Beginning #15 TCP 100.64.0.2:60076 -> push.services.mozilla.com.:443
2023-12-13T15:06:37.133184Z  INFO tun2socks5: Ending #11 TCP 100.64.0.2:60064 -> push.services.mozilla.com.:443 with Err(Kind(TimedOut))

  12. SajjadPourali commented on Dec 13, 2023

    @SajjadPourali
    SajjadPourali
    on Dec 13, 2023
    Collaborator

    Yeah, the following RST got removed, when I did the curl again.

    I open my browser, mozilla connects something. It constantly times out, RSTs the connections and reconnect immediately.

    https://github.com/planetoryd/misc/blob/master/timeout.pcapng

    The timeout issue is manageable because I have hard-coded a 5-second timeout (which I intend to make configurable #4). Firefox keeps the connections open for reuse, and in some cases, this can result in them remaining open for more than 30 seconds.

    Screenshot 2023-12-13 at 4 57 49 PM

  13. SajjadPourali commented on Dec 13, 2023

    @SajjadPourali
    SajjadPourali
    on Dec 13, 2023
    Collaborator

    It's not fixed. @SajjadPourali

    You can test ipstack with tun2socks5 under Linux. steps like this

    sudo apt install iperf3 dante-server
sudo systemctl stop danted

git clone https://github.com/ssrlive/tun2socks5.git
cd tun2socks5

cargo b -r

# start testing
sudo scripts/iperf3.sh

# clear up
sudo sh -c "pkill tun2socks5; pkill iperf3; pkill danted; ip link del tun0; ip netns del test"

    I'll set up a Linux machine over the weekend to test tun2socks5. In the meantime, if you could attach a .pcap file, I would be able to check it manually beforehand.

  14. SajjadPourali commented on Dec 13, 2023

    @SajjadPourali
    SajjadPourali
    on Dec 13, 2023
    Collaborator

    Yeah, the following RST got removed, when I did the curl again.

    I open my browser, mozilla connects something. It constantly times out, RSTs the connections and reconnect immediately.

    https://github.com/planetoryd/misc/blob/master/timeout.pcapng

    I just changed the API to use a custom timeout and increased the TCP and UDP timeouts to 60 and 30, respectively, in the main branch.

  15. SajjadPourali commented on Dec 13, 2023

    @SajjadPourali
    SajjadPourali
    on Dec 13, 2023
    Collaborator

    I used iperf3 on my personal machine through narrowlink's tunnel, which utilizes ipstack.

    Screenshot 2023-12-13 at 5 47 57 PM

  16. ple1n commented on Dec 14, 2023

    @ple1n
    ple1n
    on Dec 14, 2023
    Author

    image

    This time the apps actively tried to RST the connection (quite frequently). I was running rustup update, and it failed 2 times before a success. fa94dac

    active_rst.zip

    Perhaps there is some issue with protocol compliance ?

  17. SajjadPourali commented on Dec 14, 2023

    @SajjadPourali
    SajjadPourali
    on Dec 14, 2023
    Collaborator

    image

    This time the apps actively tried to RST the connection (quite frequently). I was running rustup update, and it failed 2 times before a success. fa94dac

    active_rst.zip

    Perhaps there is some issue with protocol compliance ?

    I can't guarantee that the Ipstack is fully compliant, but I am doing my best to implement it at least based on RFC 793. However, RFC 793 is missing some features (for example, it lacks TCP keep-alive functionality) and has some weaknesses. Therefore, the newer version of TCP is based on RFC 9293, which is more comprehensive and addresses the shortcomings of RFC 793.

    TCP keep-alive has not been implemented yet #8 (it is also not defined in RFC 793). The screenshot you mentioned shows an attempt to send a TCP keep-alive packet, and since it does not receive an ACK after several attempts, it resets the connection.

    Are you sure that the tool you're using to forward connections from the Ipstack is functioning properly?

  18. ple1n commented on Dec 14, 2023

    @ple1n
    ple1n
    on Dec 14, 2023
    Author

    image

    (image shows direct connection to socks5 proxy)

    It should be the keep-alive problem

  19. narrowlink
    locked and limited conversation to collaborators on Dec 14, 2023
  20. SajjadPourali
    converted this issue into a discussion #10 on Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions