[netfilter-core] memory leak with IP connection tracking and IP forwarding

[Harald Welte]

Hi all,
I'm experiencing a major memory leak if I run a router with IP connection
tracking enabled and packets larger than MTU size.

How do you observe such a memory leak? What exact values are you
looking at?

If I do a 'ping -f -s 64000 <Host B>' from host A via the router, the
router is losing more then 10 MB of memory per second.
Without connection tracking everything works fine.
My only glue so far is that the skbuff_head_cache is growing. Connection
tracking has a hook in the IP stack, where it gets hold of packets. If these
packets are fragmented (due to the packet size being larger than MTU size)
they are reassembled by ip_conntrack. So I guess the memory leak might
be somewhere in there.

Did you enable support for bridging?

There was a bug in the bridging (bridging firewall) code which made it
'forget' to drop the reference count (and thus introduced a memory
leak).

Regards,
Thomas.

[Thomas Spatzier]

Post by Harald Welte

I'm experiencing a major memory leak if I run a router with IP connection
tracking enabled and packets larger than MTU size.

How do you observe such a memory leak? What exact values are you
looking at?

On the router I issue a 'watch -n 1 cat /proc/meminfo'.
Then I do a 'ping -f -s 64000' from one host to another host via the router.
On each update of my watch above there are about 10 MB of memory
(MemFree) missing.
I also watched /proc/slabinfo and noticed the increasing size of the
skbuff_head_cache slab.

Post by Harald Welte

My only glue so far is that the skbuff_head_cache is growing. Connection
tracking has a hook in the IP stack, where it gets hold of packets. If
these packets are fragmented (due to the packet size being larger than
MTU size) they are reassembled by ip_conntrack. So I guess the memory
leak might be somewhere in there.

Did you enable support for bridging?
There was a bug in the bridging (bridging firewall) code which made it
'forget' to drop the reference count (and thus introduced a memory
leak).

CONFIG_BRIDGE is not set, i.e. bridging is not enabled.

[Harald Welte]

Post by Thomas Spatzier

Post by Harald Welte

I'm experiencing a major memory leak if I run a router with IP connection
tracking enabled and packets larger than MTU size.

How do you observe such a memory leak? What exact values are you
looking at?

On the router I issue a 'watch -n 1 cat /proc/meminfo'.
Then I do a 'ping -f -s 64000' from one host to another host via the router.
On each update of my watch above there are about 10 MB of memory
(MemFree) missing.

MemFree is not significant in this case. please refer to your favourite
linux kernel FAQ.

Post by Thomas Spatzier
I also watched /proc/slabinfo and noticed the increasing size of the
skbuff_head_cache slab.

does it increase and stay at a certain value, or increase indefinitely?
my guess is that you are observing quite normal behaviour, after getting
used, the slab cache allocates some memory which is then re-used again
and again.

Interesting is also not the sizze of the SLAB, but the number of objects
in use from a particular slab.

Post by Thomas Spatzier
Regards,
Thomas.

[Thomas Spatzier]

Post by Harald Welte

Post by Thomas Spatzier
On the router I issue a 'watch -n 1 cat /proc/meminfo'.
Then I do a 'ping -f -s 64000' from one host to another host via the
router. On each update of my watch above there are about 10 MB of memory
(MemFree) missing.

MemFree is not significant in this case. please refer to your favourite
linux kernel FAQ.

Somehow it must be significant, since after a few minutes the system dies
in a out-of-memory situation:

swapper: page allocation failure. order:0, mode:0x20
0067b8f0 0011811c 0046646c 8011846e 80118630 0067b868 0067b910 00400000
00000000 00000001 00407b00 00000011
Call Trace:
<000000000013aad2>| __alloc_pages+0x30e/0x374
<000000000013ac08>| __get_free_pages+0x30/0x74
<000000000013dd2c>| cache_alloc_refill+0x354/0x6fc
<000000000013d708>| __kmalloc+0xb8/0xe0
<00000000002a8d16>| pskb_expand_head+0x5a/0x190
<00000000002c7662>| ip_forward+0xe6/0x318
<00000000002c5ae0>| ip_rcv_finish+0x22c/0x33c
<00000000002b95cc>| nf_hook_slow+0x138/0x154
<00000000002c5dfe>| ip_rcv+0x20e/0x5d0
<00000000002ae3e2>| netif_receive_skb+0x1f2/0x214
<00000000002ae49a>| process_backlog+0x96/0x190
<00000000002ae616>| net_rx_action+0x82/0x164
<000000000011cda8>| do_softirq+0x90/0x100
<000000000010dfb2>| do_call_base+0x20/0x2a
<0000000000246354>| do_IRQ+0x190/0x1c8
<000000000010e3e6>| io_return+0x0/0x10

Post by Harald Welte

Post by Thomas Spatzier
I also watched /proc/slabinfo and noticed the increasing size of the
skbuff_head_cache slab.

does it increase and stay at a certain value, or increase indefinitely?
my guess is that you are observing quite normal behaviour, after getting
used, the slab cache allocates some memory which is then re-used again
and again.

The size of the SLAB increases indefinitely. Also, if I stop the test and run
some other memory intensive task, the SLAB does not shrink --- normally
it is supposed to shrink in that case, right?

Post by Harald Welte
Interesting is also not the sizze of the SLAB, but the number of objects
in use from a particular slab.

[KOVACS Krisztian]

Hi,

Post by Thomas Spatzier
On the router I issue a 'watch -n 1 cat /proc/meminfo'.
Then I do a 'ping -f -s 64000' from one host to another host via =

the

Post by Thomas Spatzier
router. On each update of my watch above there are about 10 MB of=

memory

Post by Thomas Spatzier
(MemFree) missing.

I've tried to reproduce this with 2.4.25 and 2.6.6, but couldn't find
any leaks... Can you tell me something more specific about your kernel
config and the ruleset on the firewall? Can you reproduce the same with
ip_conntrack loaded but without any iptables rules?

--=20
Regards,
Krisztian KOVACS

-
To unsubscribe from this list: send the line "unsubscribe linux-net" in
the body of a message to ***@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html