linux networking - Bonding 802.3ad and HPE switches: ingress and egress traffic not in the same interface

I have LACP configured between my server (802.3ad, layer2, so based on source MAC and destination MAC) and my switch.

Recently I saw that ingress traffic for a network peer is using one interface (eth3) while egress, for the same network peer, is using the other interface (eth1) !?

Is it the normal behaviour ?

Looking at the kernel documentation (https://www.kernel.org/doc/Documentation/networking/bonding.txt, section xmit_hash_policy): I don't think so.
But I must admit that I'm lost, really lost...

Here is my setup:

bond configuration on my server

root@server:~# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up

MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
    Aggregator ID: 2

    Number of ports: 2
    Actor Key: 17
    Partner Key: 8
    Partner Mac Address: 2c:23:3a:6a:c5:fe

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 6

Permanent HW addr: b0:83:fe:d9:93:a0
Aggregator ID: 2
Slave queue ID: 0

Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 6
Permanent HW addr: b0:83:fe:d9:93:a2

Aggregator ID: 2
Slave queue ID: 0

Switch's configuration (HPE 5130):

 display link-aggregation load-sharing mode interface Bridge-Aggregation 8
Bridge-Aggregation8 load-sharing mode: 
Layer 2 traffic: packet type-based sharing
Layer 3 traffic: packet type-based sharing



display link-aggregation verbose Bridge-Aggregation 8 
Loadsharing Type: Shar -- Loadsharing, NonS -- Non-Loadsharing 
Port Status: S -- Selected, U -- Unselected,  
             I -- Individual, * -- Management port 
Flags:  A -- LACP_Activity, B -- LACP_Timeout, C -- Aggregation, 
        D -- Synchronization, E -- Collecting, F -- Distributing, 
        G -- Defaulted, H -- Expired 


Aggregate Interface: Bridge-Aggregation8
Aggregation Mode: Dynamic
Loadsharing Type: Shar
Management VLAN : None
System ID: 0x8000, 2c23-3a6a-c5fe
Local: 
  Port             Status  Priority Oper-Key  Flag
--------------------------------------------------------------------------------
  GE1/0/8          S       32768    8         {ABCDEF}
  GE2/0/8          S       32768    8         {ABCDEF}

Remote: 
  Actor            Partner Priority Oper-Key  SystemID               Flag   
--------------------------------------------------------------------------------
  GE1/0/8          2       255      17        0xffff, b083-fed9-93a0 {ABCDEF}
  GE2/0/8          1       255      17        0xffff, b083-fed9-93a0 {ABCDEF}

I tried to change the load-balancing mode on my switch but nothing changed.

Thanks!

Answer

With LACP or static bonding, each side decides on its own how to route traffic.

Switches usually apply a SA/DA scheme - they hash the lower three or four bits of the source and destination addresses and use that as LAG port index. Simpler switches just use the MAC addresses, more sophisticated ones IP addresses (when present) or even in conjunction with TCP/UDP ports.

The intention is to have a single flow always use the same port combinations, so frames can't get out of sequence.

Using MAC addresses alone causes all traffic between two hosts (or routers) to always use the same port combination.

Using IP addresses allows flows between routers to be distributed or enables you to select (secondary) IP addresses to optimize the flows and IP/port combinations load balance different connections automatically (though not necessarily optimally).

So, ingress traffic to the host is controlled by the switch, egress traffic is controlled by the host. This can very easily lead to different ports being used in both directions. It doesn't hurt though.