[99s-extend] Cowboy HTTPS connection memory usage

[rambocoder]

Long story short, I solved the problem by adding {max_connections, 50000}
to cowboy:start_https because it default to 1024 at
https://github.com/extend/ranch/blob/master/src/ranch_listener_sup.erl#L30

However, before I figured out that setting, I did run eprof and these are
the function calls it was spending most of it's time on


FUNCTION                                    CALLS      %   TIME  [uS /
CALLS]
--------                                    -----    ---   ----
 [----------]
dict:get_slot/2                               174   1.73   1658  [
 9.53]
dict:on_bucket/3                              171   1.77   1701  [
 9.95]
erlang:setelement/3                           684   3.23   3098  [
 4.53]
dict:store_bkt_val/3                          600   5.24   5028  [
 8.38]

Then I ran etop and it showed that ranch_acceptor:maybe_wait had the most
reductions were, so I looked at the code in that
https://github.com/extend/ranch/blob/master/src/ranch_acceptor.erl#L72 and
realized that like a newb I did not set the maximum connections for the
listener :)

Problem solved. Looks like I won't need to put HAProxy in front of Cowboy
after all.

Thank you,

rambocoder

On Fri, Dec 21, 2012 at 11:51 AM, Loïc Hoguin <essen at ninenines.eu> wrote:

> Can you try enabling eprof to see where the VM spends its time?
>
>
> On 12/21/2012 05:49 PM, rambocoder wrote:
>
>> In my preliminary testing, I used Jmeter this morning since it's an
>> easy GUI load testing app and this is what I am seeing:
>>
>> With R15B03-01 [smp:4:4] [async-threads:4] [hipe] [kernel-poll:true],
>> when I establish 1K concurrent connections via HTTPS, each connection
>> takes up about 68K of memory.
>>
>> Unfortunately, after about 1050-1200 connections, on my test server the
>> Erlang scheduler jumps to 100% CPU utilization on all 4 schedulers,
>> while up to that point the scheduler's load was oscillating up and down.
>> Using the Observer, there is only 1 ssl_connection_sup  in the ssl
>> application, having to deal with 1000+ gen_fsm workers, so that might be
>> the bottleneck. Since the ulimit on my server is 50000 I don't think I
>> am hitting any type of file handler's limit.
>>
>> Loïc and the group, am I missing some setting that is causing the
>> scheduler to go to 100% CPU and the run que in observer to be 99?
>>
>> Sincerely,
>>
>> rambocoder
>>
>>
>>
>> On Fri, Dec 21, 2012 at 6:45 AM, Loïc Hoguin <essen at ninenines.eu
>> <mailto:essen at ninenines.eu>> wrote:
>>
>>     On 12/21/2012 04:34 AM, rambocoder wrote:
>>
>>         Does anybody know either from benchmarks or real world data what
>>         is the
>>         average memory footprint of each concurrent HTTPS connection to
>>         cowboy?
>>
>>
>>     I don't have anything, sorry. I'm guessing it consumes a lot more
>>     than TCP though.
>>
>>
>>         SSL app in Erlang reuses SSL session-ids so I am not sure if the
>>         Apache
>>         Bench I test with reuses the session id or it does not.
>>
>>
>>     I wouldn't know, but I wouldn't trust Apache Bench doing the right
>>     thing. Any other benchmark tool usually works better in my experience.
>>
>>
>>         BTW, what makes an erlang api "documented" vs "undocumented". For
>>         example ssl:session_info/1 function here (
>>         https://github.com/erlang/otp/**__blob/maint/lib/ssl/src/ssl._**
>> _erl#L411<https://github.com/erlang/otp/__blob/maint/lib/ssl/src/ssl.__erl#L411>
>>
>>         <https://github.com/erlang/**otp/blob/maint/lib/ssl/src/**
>> ssl.erl#L411<https://github.com/erlang/otp/blob/maint/lib/ssl/src/ssl.erl#L411>
>> >
>>         ) has
>>         a spec and a short doc, but session_info is not described
>>         http://www.erlang.org/doc/man/**__ssl.html<http://www.erlang.org/doc/man/__ssl.html>
>>
>>         <http://www.erlang.org/doc/**man/ssl.html<http://www.erlang.org/doc/man/ssl.html>>
>> .ssl:session_info/1 is
>>         a useful
>>         function to be able to track if the load generator is reusing
>>         the SSL
>>         session_id or it is generating new one, because that would make
>>         all the
>>         difference during measurement due to Erlang caching SSL sessions
>>         by default.
>>
>>
>>     The documentation is separate (they're not using edoc). It's perhaps
>>     not deemed useful enough for documenting it. I wouldn't worry about
>>     using it for measurements though.
>>
>>     Try asking Ingela on the ML about it, perhaps they just forgot to
>>     document it.
>>
>>     --
>>     Loďc Hoguin
>>     Erlang Cowboy
>>     Nine Nines
>>     http://ninenines.eu
>>
>>
>>
>>
>> ______________________________**_________________
>> Extend mailing list
>> Extend at lists.ninenines.eu
>> http://lists.ninenines.eu:81/**listinfo/extend<http://lists.ninenines.eu:81/listinfo/extend>
>>
>>
>
> --
> Loïc Hoguin
>
> Erlang Cowboy
> Nine Nines
> http://ninenines.eu
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ninenines.eu/archives/extend/attachments/20121221/945f636e/attachment.html>