diff options
| author | David S. Miller <davem@davemloft.net> | 2018-12-03 15:58:33 -0800 |
|---|---|---|
| committer | David S. Miller <davem@davemloft.net> | 2018-12-03 15:58:59 -0800 |
| commit | 6e360f7331132822f096399a54f40bc61887aa3c (patch) | |
| tree | c8c872ef216758c80a3655edb104f00903eced8c /include/linux | |
| parent | ce01a56ba3d9a56e9c7dd4662e2753b102a17d62 (diff) | |
| parent | db63e489c7aa03bab86960d6bcb6bf1fad8eea4e (diff) | |
| download | cachepc-linux-6e360f7331132822f096399a54f40bc61887aa3c.tar.gz cachepc-linux-6e360f7331132822f096399a54f40bc61887aa3c.zip | |
Merge branch 'udp-msg_zerocopy'
Willem de Bruijn says:
====================
udp msg_zerocopy
Enable MSG_ZEROCOPY for udp sockets
Patch 1/3 is the main patch, a rework of RFC patch
http://patchwork.ozlabs.org/patch/899630/
more details in the patch commit message
Patch 2/3 is an optimization to remove a branch from the UDP hot path
and refcount_inc/refcount_dec_and_test pair when zerocopy is used.
This used to be included in the first patch in v2.
Patch 3/3 runs the already existing udp zerocopy tests
as part of kselftest
See also recent Linux Plumbers presentation
https://linuxplumbersconf.org/event/2/contributions/106/attachments/104/128/willemdebruijn-lpc2018-udpgso-presentation-20181113.pdf
Changes:
v1 -> v2
- Fixup reverse christmas tree violation
v2 -> v3
- Split refcount avoidance optimization into separate patch
- Fix refcount leak on error in fragmented case
(thanks to Paolo Abeni for pointing this one out!)
- Fix refcount inc on zero
v3 -> v4
- Move skb_zcopy_set below the only kfree_skb that might cause
a premature uarg destroy before skb_zerocopy_put_abort
- Move the entire skb_shinfo assignment block, to keep that
cacheline access in one place
====================
Signed-off-by: David S. Miller <davem@davemloft.net>
Diffstat (limited to 'include/linux')
| -rw-r--r-- | include/linux/skbuff.h | 13 |
1 files changed, 9 insertions, 4 deletions
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 73902acf2b71..75d50ab7997c 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -481,10 +481,11 @@ static inline void sock_zerocopy_get(struct ubuf_info *uarg) } void sock_zerocopy_put(struct ubuf_info *uarg); -void sock_zerocopy_put_abort(struct ubuf_info *uarg); +void sock_zerocopy_put_abort(struct ubuf_info *uarg, bool have_uref); void sock_zerocopy_callback(struct ubuf_info *uarg, bool success); +int skb_zerocopy_iter_dgram(struct sk_buff *skb, struct msghdr *msg, int len); int skb_zerocopy_iter_stream(struct sock *sk, struct sk_buff *skb, struct msghdr *msg, int len, struct ubuf_info *uarg); @@ -1325,10 +1326,14 @@ static inline struct ubuf_info *skb_zcopy(struct sk_buff *skb) return is_zcopy ? skb_uarg(skb) : NULL; } -static inline void skb_zcopy_set(struct sk_buff *skb, struct ubuf_info *uarg) +static inline void skb_zcopy_set(struct sk_buff *skb, struct ubuf_info *uarg, + bool *have_ref) { if (skb && uarg && !skb_zcopy(skb)) { - sock_zerocopy_get(uarg); + if (unlikely(have_ref && *have_ref)) + *have_ref = false; + else + sock_zerocopy_get(uarg); skb_shinfo(skb)->destructor_arg = uarg; skb_shinfo(skb)->tx_flags |= SKBTX_ZEROCOPY_FRAG; } @@ -1373,7 +1378,7 @@ static inline void skb_zcopy_abort(struct sk_buff *skb) struct ubuf_info *uarg = skb_zcopy(skb); if (uarg) { - sock_zerocopy_put_abort(uarg); + sock_zerocopy_put_abort(uarg, false); skb_shinfo(skb)->tx_flags &= ~SKBTX_ZEROCOPY_FRAG; } } |