Hi.

I found a performance problems with HTTPClient and HTTPText.

The library is great. I tried to read a text file from a HTTP server.

But the performance has characteristic abnormality.

So I check the performace problems.

Please see the result.

The tests

Text files on a HTTP server

I created some text files on a HTTP server.

The size are 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096 bytes.

e.g.) http://mbed.org/media/uploads/shintamainjp/textfile_32bytes.txt

A test program

1. Read a text file from HTTP server.
2. Buffer size of HTTPText : 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096
3. File size on HTTP server : 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096

e.g.) http://mbed.org/users/shintamainjp/programs/PerformanceProblemsWithHTTPText/latest

The results

(Buffer size of a HTTPText : 8)
+---------+--------+----+
|File size|Time[ms]|Code|
+---------+--------+----+
|        8|   31525| 200|
|       16|   30976| 200|
|       32|   30974| 200|
|       64|   30976| 200|
|      128|   30978| 200|
|      256|   30974| 200|
|      512|   30971| 200|
|     1024|   30974| 200|
|     2048|   30975| 200|
|     4096|   30976| 200|
+---------+--------+----+

(Buffer size of a HTTPText : 16)
+---------+--------+----+
|File size|Time[ms]|Code|
+---------+--------+----+
|        8|     831| 200|
|       16|   30977| 200|
|       32|   30970| 200|
|       64|   30977| 200|
|      128|   30975| 200|
|      256|   30978| 200|
|      512|   30972| 200|
|     1024|   30976| 200|
|     2048|   30976| 200|
|     4096|   30976| 200|
+---------+--------+----+

(Buffer size of a HTTPText : 32)
+---------+--------+----+
|File size|Time[ms]|Code|
+---------+--------+----+
|        8|     831| 200|
|       16|     971| 200|
|       32|   30975| 200|
|       64|   30976| 200|
|      128|   30980| 200|
|      256|   30972| 200|
|      512|   30976| 200|
|     1024|   30985| 200|
|     2048|   30966| 200|
|     4096|   30974| 200|
+---------+--------+----+

(Buffer size of a HTTPText : 64)
+---------+--------+----+
|File size|Time[ms]|Code|
+---------+--------+----+
|        8|     830| 200|
|       16|     975| 200|
|       32|     975| 200|
|       64|   30971| 200|
|      128|   30980| 200|
|      256|   30972| 200|
|      512|   30980| 200|
|     1024|   30970| 200|
|     2048|   30976| 200|
|     4096|   30975| 200|
+---------+--------+----+

(Buffer size of a HTTPText : 128)
+---------+--------+----+
|File size|Time[ms]|Code|
+---------+--------+----+
|        8|     828| 200|
|       16|     975| 200|
|       32|     974| 200|
|       64|     976| 200|
|      128|   30974| 200|
|      256|   30976| 200|
|      512|   30976| 200|
|     1024|   30975| 200|
|     2048|   30976| 200|
|     4096|   30974| 200|
+---------+--------+----+

(Buffer size of a HTTPText : 256)
+---------+--------+----+
|File size|Time[ms]|Code|
+---------+--------+----+
|        8|     829| 200|
|       16|     974| 200|
|       32|     973| 200|
|       64|     974| 200|
|      128|     974| 200|
|      256|   30975| 200|
|      512|   30977| 200|
|     1024|   30975| 200|
|     2048|   30977| 200|
|     4096|   30977| 200|
+---------+--------+----+

(Buffer size of a HTTPText : 512)
+---------+--------+----+
|File size|Time[ms]|Code|
+---------+--------+----+
|        8|     827| 200|
|       16|     975| 200|
|       32|     973| 200|
|       64|     973| 200|
|      128|     978| 200|
|      256|     969| 200|
|      512|   30975| 200|
|     1024|   30977| 200|
|     2048|   30975| 200|
|     4096|   30975| 200|
+---------+--------+----+

(Buffer size of a HTTPText : 1024)
+---------+--------+----+
|File size|Time[ms]|Code|
+---------+--------+----+
|        8|     829| 200|
|       16|     973| 200|
|       32|     973| 200|
|       64|     975| 200|
|      128|     974| 200|
|      256|     974| 200|
|      512|     975| 200|
|     1024|   30974| 200|
|     2048|   30976| 200|
|     4096|   30976| 200|
+---------+--------+----+

(Buffer size of a HTTPText : 2048)
+---------+--------+----+
|File size|Time[ms]|Code|
+---------+--------+----+
|        8|     830| 200|
|       16|     974| 200|
|       32|     977| 200|
|       64|     976| 200|
|      128|     971| 200|
|      256|     976| 200|
|      512|    1106| 200|
|     1024|     848| 200|
|     2048|   31364| 200|
|     4096|   30851| 200|
+---------+--------+----+

(Buffer size of a HTTPText : 4096)
+---------+--------+----+
|File size|Time[ms]|Code|
+---------+--------+----+
|        8|     564| 200|
|       16|     968| 200|
|       32|     973| 200|
|       64|     974| 200|
|      128|     975| 200|
|      256|     975| 200|
|      512|     971| 200|
|     1024|     976| 200|
|     2048|    1244| 200|
|     4096|   31251| 200|
+---------+--------+----+

Why it's spent over 30 seconds?

I think the library has a performance problems in that implementations.

I can't see the sources. so I don't know why it's spent so much time.

But I wanna be fix it. I can't wait over 30 seconds for getting a data from HTTP server.

The network function is big feauture in mbed I think.

Please check the implementations.

Resources

* The test program : PerformanceProblemsWithHTTPText

Shinichiro Nakamura wrote:
I can't see the sources

Hi Shinichiro, try here http://mbed.org/users/donatien/programs/NetServicesSource/5zh9t/docs/files.html for the source code.

Was the data actually arriving in 30 seconds, or was that the timeout?

Regards
Daniel

Mr. Daniel Peter-san

Thank you for your reply.
You are right.

I tried to change the timeout with HTTPClient::setTimeout() function.
It depend on that timeout time.

The result of the test [ client.setTimeout(10000); ]

(Buffer size of a HTTPText : 8)
+---------+--------+----+
|File size|Time[ms]|Code|
+---------+--------+----+
|        8|   11522| 200|
|       16|   10972| 200|
|       32|   10973| 200|
|       64|   10976| 200|
|      128|   10974| 200|
|      256|   10975| 200|
|      512|   10976| 200|
|     1024|   10973| 200|
|     2048|   10977| 200|
|     4096|   10972| 200|
+---------+--------+----+

(Buffer size of a HTTPText : 16)
+---------+--------+----+
|File size|Time[ms]|Code|
+---------+--------+----+
|        8|     831| 200|
|       16|   10974| 200|
|       32|   10979| 200|
|       64|   10969| 200|
|      128|   10976| 200|
|      256|   10976| 200|
|      512|   10975| 200|
|     1024|   10975| 200|
|     2048|   10976| 200|
|     4096|   10970| 200|
+---------+--------+----+

Well, the return code (200) isn't right for users.
It should be indicate the timeout (HTTP_TIMEOUT) if timeout is true.

But the timeout isn't server side timeout.
It's just client  (in the API) reason.
So it not indicate the timeout status I think.

Anyway, Thank you so much Mr. Daniel Peter-san.

Best regards,

Shin. :)

Dear Shinichiro-san

There are two return codes with the get method. Firstly, the get method itself returns an HTTPResult, after the get returns you can then call getHTTPResponseCode().

If you check, you might find that the HTTPResult is an 8 (HTTP_TIMEOUT), and the HTTP response code is just defaulting to 200.

Here is a code snippet from the pachube client wrapper I wrote (admittedly with a post rather than a get, but the idea is the same):

    _result = _client.post(uri.c_str(), _csvContent, NULL);
    _response = _client.getHTTPResponseCode();

I just took a look at your test program as well. You are getting the HTTPResult into the variable r, but then ignoring it. Perhaps you want to add it as an extra column in the program print out?

Regards
Daniel

Dear Mr.Daniel Peter-san

You are getting the HTTPResult into the variable r, but then ignoring it. Perhaps you want to add it as an extra column in the program print out?

Oops. :-o

Sorry. I fixed it. PerformanceProblemsWithHTTPText

(Buffer size of a HTTPText : 1024)
+---------+--------+------+--------+
|File size|Time[ms]|Result|Response|
+---------+--------+------+--------+
|        8|     769|     0|     200|
|       16|     963|     0|     200|
|       32|     965|     0|     200|
|       64|     962|     0|     200|
|      128|     962|     0|     200|
|      256|     963|     0|     200|
|      512|     964|     0|     200|
|     1024|    1964|     8|     200|
|     2048|    1963|     8|     200|
|     4096|    1962|     8|     200|
+---------+--------+------+--------+

(Buffer size of a HTTPText : 2048)
+---------+--------+------+--------+
|File size|Time[ms]|Result|Response|
+---------+--------+------+--------+
|        8|     779|     0|     200|
|       16|     956|     0|     200|
|       32|     963|     0|     200|
|       64|     964|     0|     200|
|      128|     960|     0|     200|
|      256|     964|     0|     200|
|      512|     971|     0|     200|
|     1024|     956|     0|     200|
|     2048|    2224|     8|     200|
|     4096|    1969|     8|     200|
+---------+--------+------+--------+

Ahh. It indicate HTTP_TIMEOUT.

Unn...

It indicate HTTP_TIMEOUT when the buffer size is smaller than a file size on a server. :(

So how can I get a big file from a server? Current API can't get it from a server I think.

From my point of view, It need a 'callback' based function without big buffer.

Regards

Shin.

Hi Shin

Instead of using an HTTPText object, Donatien wrote the HTTPStream class exactly for this purpose eg. for getting mp3 data from a server (or big files in small pieces).

Donatien's sample is here, also Elliot wrote a weather page scraper here.

Hope that helps!

Regards
Daniel

(Edited - changed HTTPClient to HTTPText in first line)

Hi Daniel

Oh! The class parent of HTTPText and HTTPStream is HTTPData. :)

I got it.

Thank you so much Mr.Daniel.

Regards

Shin. :)