本文主要是在报文在OVS的转发路径上都耗时多长。

物理主机发包流程

首先要知道转发路径经过了那些函数,才方便找到抓取点。

vhost_worker->handle_tx_kick->handle_tx(sock->ops->sendmsg)->tun_sendmsg->tun_get_user->netif_rx_ni->do_softirq->call_softirq->__do_softirq->net_rx_action->process_backlog->__netif_receive_skb->__netif_receive_skb_core->netdev_frame_hook->netdev_port_receive->ovs_vport_receive->ovs_dp_process_packet->ovs_execute_actions->do_execute_actions->do_output->ovs_vport_send->vxlan_tnl_send->vxlan_xmit_skb->udp_tunnel_xmit_skb->iptunnel_xmit->ip_local_out_sk->__ip_local_out->__ip_local_out_sk->dst_output_sk->ip_output->ip_finish_output->ip_finish_output2->neigh_hh_output->dev_queue_xmit->__dev_xmit_skb->sch_direct_xmit->dev_hard_start_xmit->xmit_one->netdev_start_xmit->__netdev_start_xmit->bond_start_xmit->__bond_start_xmit->bond_3ad_xor_xmit->bond_dev_queue_xmit->dev_queue_xmit->__dev_xmit_skb->sch_direct_xmit->dev_hard_start_xmit->xmit_one->netdev_start_xmit->__netdev_start_xmit->ixgbe_xmit_frame

时间记录点

netif_rx_ni 第一个记录点,vhost的获取到skb之后,第一个接口。
ip_local_out_sk 第二个记录点,报文从vhost到ovs查表并且vxlan封装完毕准备发送的时间。
dev_queue_xmit 第三个记录点,报文经过netfilter、route和邻居子系统2层封装。
dev_queue_xmit 第四个记录点,报文经过bond的时间。

抓取结果

64k大包

[365665.609813] netif_rx_ni:           skb ffff881f7b117700 len 65212
[365665.609818] ip_local_out_sk:       skb ffff881f7b117700 len 65262
[365665.609825] dev_queue_xmit:       skb ffff881f7b117700 len 65276
[365665.609831] dev_queue_xmit:       skb ffff881f7b117700 len 65276

[365665.610047] netif_rx_ni:           skb ffff881fcf9ad000 len 65212
[365665.610051] ip_local_out_sk:       skb ffff881fcf9ad000 len 65262
[365665.610058] dev_queue_xmit:       skb ffff881fcf9ad000 len 65276
[365665.610062] dev_queue_xmit:       skb ffff881fcf9ad000 len 65276

0.234ms包间距,0.018ms包路径
(65212 - 40) * 8 / (365665.610047 - 365665.609813) = 2228102564.102564 bps = 2.07 Gbps

1500包

[367138.736316] netif_rx_ni:           skb ffff881f85e1e800 len 1500
[367138.736319] ip_local_out_sk:       skb ffff881f85e1e800 len 1550
[367138.736320] dev_queue_xmit:       skb ffff881f85e1e800 len 1564
[367138.736321] dev_queue_xmit:       skb ffff881f85e1e800 len 1564

[367138.736325] netif_rx_ni:           skb ffff881f85e1ee00 len 1500
[367138.736327] ip_local_out_sk:       skb ffff881f85e1ee00 len 1550
[367138.736330] dev_queue_xmit:       skb ffff881f85e1ee00 len 1564
[367138.736332] dev_queue_xmit:       skb ffff881f85e1ee00 len 1564

0.009ms包间距,0.005ms包路径
(1500 - 40) * 8 / (367138.736325 - 367138.736316) = 1297777777.777778 bps = 1.2 Gbps

可能是UDP大包

[433171.804667] netif_rx_ni:            skb ffff881f896e2e00 len 8868
[433171.804673] ip_local_out_sk:        skb ffff881f896e2e00 len 8918
[433171.804676] dev_queue_xmit:    skb ffff881f896e2e00 len 8932
[433171.804678] dev_queue_xmit:    skb ffff881f896e2e00 len 8932

[433171.804765] netif_rx_ni:            skb ffff881f88948200 len 8868
[433171.804768] ip_local_out_sk:        skb ffff881f88948200 len 8918
[433171.804770] dev_queue_xmit:    skb ffff881f88948200 len 8932
[433171.804772] dev_queue_xmit:    skb ffff881f88948200 len 8932

[433171.805772] netif_rx_ni:            skb ffff881f852a2f00 len 8868
[433171.805775] ip_local_out_sk:        skb ffff881f852a2f00 len 8918
[433171.805778] dev_queue_xmit:    skb ffff881f852a2f00 len 8932
[433171.805780] dev_queue_xmit:    skb ffff881f852a2f00 len 8932

0.098ms包间距,0.011ms包路径
(8868 - 40) * 8 / (433171.804765 - 433171.804667) = 720653061.2244898 bps = 687Mbps 实测120Mbps
(8868 - 40) 2 8 / (433171.805772 - 433171.804667) = 127826244.3438914 bps = 122Mbps

可能是UDP小包

[433389.227051] netif_rx_ni:            skb ffff881fcee54500 len 128
[433389.227053] ip_local_out_sk:        skb ffff881fcee54500 len 178
[433389.227055] dev_queue_xmit:    skb ffff881fcee54500 len 192
[433389.227056] dev_queue_xmit:    skb ffff881fcee54500 len 192

[433389.227059] netif_rx_ni:            skb ffff881fcee54400 len 128
[433389.227061] ip_local_out_sk:        skb ffff881fcee54400 len 178
[433389.227063] dev_queue_xmit:    skb ffff881fcee54400 len 192
[433389.227064] dev_queue_xmit:    skb ffff881fcee54400 len 192

[433389.227067] netif_rx_ni:            skb ffff881fcee55a00 len 128
[433389.227069] ip_local_out_sk:        skb ffff881fcee55a00 len 178
[433389.227071] dev_queue_xmit:    skb ffff881fcee55a00 len 192
[433389.227072] dev_queue_xmit:    skb ffff881fcee55a00 len 192

[433389.227076] netif_rx_ni:            skb ffff881fcee54700 len 128
[433389.227078] ip_local_out_sk:        skb ffff881fcee54700 len 178
[433389.227079] dev_queue_xmit:    skb ffff881fcee54700 len 192
[433389.227081] dev_queue_xmit:    skb ffff881fcee54700 len 192

[433389.227084] netif_rx_ni:            skb ffff881fcee54300 len 128
[433389.227085] ip_local_out_sk:        skb ffff881fcee54300 len 178
[433389.227087] dev_queue_xmit:    skb ffff881fcee54300 len 192
[433389.227088] dev_queue_xmit:    skb ffff881fcee54300 len 192

[433389.227091] netif_rx_ni:            skb ffff881fcee54a00 len 128
[433389.227093] ip_local_out_sk:        skb ffff881fcee54a00 len 178
[433389.227095] dev_queue_xmit:    skb ffff881fcee54a00 len 192
[433389.227096] dev_queue_xmit:    skb ffff881fcee54a00 len 192

[433389.227098] netif_rx_ni:            skb ffff881fcee54c00 len 128
[433389.227100] ip_local_out_sk:        skb ffff881fcee54c00 len 178
[433389.227102] dev_queue_xmit:    skb ffff881fcee54c00 len 192
[433389.227103] dev_queue_xmit:    skb ffff881fcee54c00 len 192

[433389.227106] netif_rx_ni:            skb ffff881fd191c300 len 128
[433389.227108] ip_local_out_sk:        skb ffff881fd191c300 len 178
[433389.227110] dev_queue_xmit:    skb ffff881fd191c300 len 192
[433389.227111] dev_queue_xmit:    skb ffff881fd191c300 len 192

[433389.227114] netif_rx_ni:            skb ffff881fd191dd00 len 128
[433389.227116] ip_local_out_sk:        skb ffff881fd191dd00 len 178
[433389.227118] dev_queue_xmit:    skb ffff881fd191dd00 len 192
[433389.227119] dev_queue_xmit:    skb ffff881fd191dd00 len 192

[433389.227121] netif_rx_ni:            skb ffff881fd191d000 len 128
[433389.227124] ip_local_out_sk:        skb ffff881fd191d000 len 178
[433389.227127] dev_queue_xmit:    skb ffff881fd191d000 len 192
[433389.227129] dev_queue_xmit:    skb ffff881fd191d000 len 192

[433389.227147] netif_rx_ni:            skb ffff881fcedbc200 len 128
[433389.227149] ip_local_out_sk:        skb ffff881fcedbc200 len 178
[433389.227151] dev_queue_xmit:    skb ffff881fcedbc200 len 192
[433389.227152] dev_queue_xmit:    skb ffff881fcedbc200 len 192

第二个轮回
[433389.227259] netif_rx_ni:            skb ffff881fcedbc500 len 128
[433389.227261] ip_local_out_sk:        skb ffff881fcedbc500 len 178
[433389.227263] dev_queue_xmit:    skb ffff881fcedbc500 len 192
[433389.227264] dev_queue_xmit:    skb ffff881fcedbc500 len 192

第三个轮回
[433389.227329] netif_rx_ni:            skb ffff881fcedbc700 len 128
[433389.227331] ip_local_out_sk:        skb ffff881fcedbc700 len 178
[433389.227332] dev_queue_xmit:    skb ffff881fcedbc700 len 192
[433389.227334] dev_queue_xmit:    skb ffff881fcedbc700 len 192

第四个轮回
[433389.227448] netif_rx_ni:            skb ffff881fcedbd400 len 128
[433389.227450] ip_local_out_sk:        skb ffff881fcedbd400 len 178
[433389.227452] dev_queue_xmit:    skb ffff881fcedbd400 len 192
[433389.227454] dev_queue_xmit:    skb ffff881fcedbd400 len 192

0.008ms包间距,0.005ms包路径
(128 - 40) * 8 / (433389.227059 - 433389.227051) = 88000000 bps = 84Mbps 实测51Mbps多一些,所以不准,发现每10个包会出现一段时间增常,所以后来增加了包的统计。
第一轮回
(128 - 40) 10 8 / (433389.227147 - 433389.227051) = 73333333.33333333 bps = 70Mbps
第四轮回
(128 - 40) 10 8 / (433389.227448 - 433389.227329) = 59159663.86554622 bps = 56Mbps

最后修改:2021 年 08 月 18 日 08 : 44 PM