Visit Open-E website
Results 1 to 8 of 8

Thread: 10gb sysctl tuneing

  1. #1
    Join Date
    Oct 2006
    Posts
    202

    Default 10gb sysctl tuneing

    Just wanting to know if anyone has or can provide a good set of values as a starting point for tuning 10GB. I am only getting a third of the expected throughput. Does the sysctl changes effect iscsi and replication?

    Thanks

  2. #2
    Join Date
    Oct 2010
    Location
    GA
    Posts
    935

    Default

    For the replication link, you can see here: http://kb.open-e.com/Tuning-recommen...1GbE_1603.html
    This is effective for 10GBe as well.

  3. #3
    Join Date
    Oct 2010
    Location
    GA
    Posts
    935

    Default

    Sysctl can be used to tune some TCP parameters, check your inbox for that small update too

  4. #4
    Join Date
    Oct 2006
    Posts
    202

    Default

    Thanks for this, if anyone out there have a suggestions for for values that are a good starting point would be appreciated I have found the following information not sure if it is any good or not, maybe someone with a good knowledge of tuning may be able to offer some advice.

    ###
    ### TUNING NETWORK PERFORMANCE ###
    ###

    # Do a 'modprobe tcp_cubic' first
    net.ipv4.tcp_congestion_control = cubic

    # Turn on the tcp_window_scaling
    net.ipv4.tcp_window_scaling = 1

    # Increase the maximum total buffer-space allocatable
    # This is measured in units of pages (4096 bytes)
    net.ipv4.tcp_mem = 65536 131072 262144
    net.ipv4.udp_mem = 65536 131072 262144

    # Increase the read-buffer space allocatable
    net.ipv4.tcp_rmem = 8192 87380 16777216
    net.ipv4.udp_rmem_min = 16384
    net.core.rmem_default = 131072
    net.core.rmem_max = 16777216

    # Increase the write-buffer-space allocatable
    net.ipv4.tcp_wmem = 8192 65536 16777216
    net.ipv4.udp_wmem_min = 16384
    net.core.wmem_default = 131072
    net.core.wmem_max = 16777216

    # Increase number of incoming connections
    net.core.somaxconn = 32768

    # Increase number of incoming connections backlog
    net.core.netdev_max_backlog = 4096
    net.core.dev_weight = 64

    # Increase the maximum amount of option memory buffers
    net.core.optmem_max = 65536

    # Increase the maximum number of skb-heads to be cached
    #net.core.hot_list_length = 1024

    # Increase the tcp-time-wait buckets pool size to prevent simple DOS attacks
    net.ipv4.tcp_max_tw_buckets = 1440000
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_tw_reuse = 1

    # Limit number of orphans, each orphan can eat up to 16M (max wmem) of unswappable memory
    net.ipv4.tcp_max_orphans = 16384
    net.ipv4.tcp_orphan_retries = 0

    # Increase the maximum memory used to reassemble IP fragments
    net.ipv4.ipfrag_high_thresh = 512000
    net.ipv4.ipfrag_low_thresh = 446464

    # don't cache ssthresh from previous connection
    net.ipv4.tcp_no_metrics_save = 1
    net.ipv4.tcp_moderate_rcvbuf = 1

    # Increase RPC slots
    sunrpc.tcp_slot_table_entries = 32
    sunrpc.udp_slot_table_entries = 32

    # Increase size of RPC datagram queue length
    net.unix.max_dgram_qlen = 50

    # Don't allow the arp table to become bigger than this
    net.ipv4.neigh.default.gc_thresh3 = 2048

    # Tell the gc when to become aggressive with arp table cleaning.
    # Adjust this based on size of the LAN. 1024 is suitable for most /24 networks
    net.ipv4.neigh.default.gc_thresh2 = 1024

    # Adjust where the gc will leave arp table alone - set to 32.
    net.ipv4.neigh.default.gc_thresh1 = 32

    # Adjust to arp table gc to clean-up more often
    net.ipv4.neigh.default.gc_interval = 30

    # Increase TCP queue length
    net.ipv4.neigh.default.proxy_qlen = 96
    net.ipv4.neigh.default.unres_qlen = 6

    # Enable Explicit Congestion Notification (RFC 3168), disable it if it doesn't work for you
    net.ipv4.tcp_ecn = 1
    net.ipv4.tcp_ecn = 2
    net.ipv4.tcp_reordering = 3

    # How many times to retry killing an alive TCP connection
    net.ipv4.tcp_retries2 = 15
    net.ipv4.tcp_retries1 = 3

    # This will enusre that immediatly subsequent connections use the new values
    net.ipv4.route.flush = 1
    net.ipv6.route.flush = 1

  5. #5

    Default

    Hi !

    We are also not getting the performance we expected from our NFS Cluster running
    on 10GbE and an pure SSD raid system.
    The Disk Performance Test from Console delivers about 3.5 GB/s read speed.
    The maximum NFS speed we can get through an LACP bond with two 10GbE interfaces
    is about 500MB/s, which is obviously to slow.
    The systems are running at DSSV7 7.0up54.9101.18255.

    Does the sysctl small update apply to this version too ?
    And what would be the parameters to be tuned ?


    Kind regards,
    Ralph

  6. #6

    Default

    Yes this build will work with the Sysctl tool functions small update. Being that there 100's of options to adjust the only
    way would be to review IPv4 variables from the link below and test, there might be other sites providing additional information as well.

    https://www.kernel.org/doc/Documenta.../ip-sysctl.txt
    All the best,

    Todd Maxwell


    Follow the red "E"
    Facebook | Twitter | YouTube

  7. #7
    Join Date
    Oct 2006
    Posts
    202

    Default

    Quote Originally Posted by DIESOFTWARE-RS View Post
    Hi !

    We are also not getting the performance we expected from our NFS Cluster running
    on 10GbE and an pure SSD raid system.
    The Disk Performance Test from Console delivers about 3.5 GB/s read speed.
    The maximum NFS speed we can get through an LACP bond with two 10GbE interfaces
    is about 500MB/s, which is obviously to slow.
    The systems are running at DSSV7 7.0up54.9101.18255.

    Does the sysctl small update apply to this version too ?
    And what would be the parameters to be tuned ?


    Kind regards,
    Ralph
    Is it only since going to the latest version you are experiencing issues?

    Gavin

  8. #8

    Default

    Hi Gavin,

    can't answer that, the systems were build new with this version.

    Kind regards
    Ralph

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •