Skip to content

Commit

Permalink
Various changes. See README for details
Browse files Browse the repository at this point in the history
  • Loading branch information
Chandrashekhar Mullaparthi committed Sep 22, 2010
1 parent c6a698c commit d756a2b
Show file tree
Hide file tree
Showing 7 changed files with 319 additions and 109 deletions.
26 changes: 23 additions & 3 deletions README
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,32 @@ ibrowse is available under two different licenses. LGPL or the BSD license.

Comments to : Chandrashekhar.Mullaparthi@gmail.com

Version : 1.6.2
Version : 2.0.0

Latest version : git://github.com/cmullaparthi/ibrowse.git

CONTRIBUTIONS & CHANGE HISTORY
==============================
22-09-2010 - * Added option preserve_chunked_encoding. This allows the caller to get
the raw HTTP response when the Transfer-Encoding is Chunked. This feature
was requested by Benoit Chesneau who wanted to write a HTTP proxy using
ibrowse.
* Fixed bug with the {stream_to, {Pid, once}} option. Bug report and lot
of help from Filipe David Manana. Thank you Filipe.
* The {error, conn_failed} and {error, send_failed} return values are
now of the form {error, {conn_failed, Err}} and
{error, {send_failed, Err}}. This is so that the specific socket error
can be returned to the caller. I think it looks a bit ugly, but that
is the best compromise I could come up with.
* Added application configuration parameters default_max_sessions and
default_max_pipeline_size. These were previously hard coded to 10.
* Versioning of ibrowse now follows the Semantic Versioning principles.
See http://semver.org. Thanks to Anthony Molinaro for nudging me in
this direction.
* The connect_timeout option now only applies to the connection setup
phase. In previous versions, the time taken to setup the connection
was deducted from the specified timeout value for the request.

17-07-2010 - * Merged change made by Filipe David Manana to use the base64
module for encoding/decoding.

Expand Down Expand Up @@ -153,7 +173,7 @@ CONTRIBUTIONS & CHANGE HISTORY
12-01-2007 - Derek Upham sent in a bug fix. The reset_state function was not
behaving correctly when the transfer encoding was not chunked.

13-11-2006 - Youn�s Hafri reported a bug where ibrowse was not returning the
13-11-2006 - Youns Hafri reported a bug where ibrowse was not returning the
temporary filename when the server was closing the connection
after sending the data (as in HTTP/1.0).
Released ibrowse under the BSD license
Expand All @@ -172,7 +192,7 @@ CONTRIBUTIONS & CHANGE HISTORY
22-Nov-2005 - Added ability to generate requests using the Chunked
Transfer-Encoding.

08-May-2005 - Youn�s Hafri made a CRUX LINUX port of ibrowse.
08-May-2005 - Youns Hafri made a CRUX LINUX port of ibrowse.
http://yhafri.club.fr/crux/index.html

Here are some usage examples. Enjoy!
Expand Down
11 changes: 8 additions & 3 deletions doc/ibrowse.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ <h1>Module ibrowse</h1>
<ul class="index"><li><a href="#description">Description</a></li><li><a href="#index">Function Index</a></li><li><a href="#functions">Function Details</a></li></ul>The ibrowse application implements an HTTP 1.1 client.
<p>Copyright © 2005-2010 Chandrashekhar Mullaparthi</p>

<p><b>Version:</b> 1.6.0</p>
<p><b>Version:</b> 2.0.0</p>
<p><b>Behaviours:</b> <a href="gen_server.html"><tt>gen_server</tt></a>.</p>
<p><b>Authors:</b> Chandrashekhar Mullaparthi (<a href="mailto:chandrashekhar dot mullaparthi at gmail dot com"><tt>chandrashekhar dot mullaparthi at gmail dot com</tt></a>).</p>

Expand Down Expand Up @@ -202,7 +202,7 @@ <h3 class="function"><a name="send_req-5">send_req/5</a></h3>
<div class="spec">
<p><tt>send_req(Url::string(), Headers::<a href="#type-headerList">headerList()</a>, Method::<a href="#type-method">method()</a>, Body::<a href="#type-body">body()</a>, Options::<a href="#type-optionList">optionList()</a>) -&gt; <a href="#type-response">response()</a></tt>
<ul class="definitions"><li><tt><a name="type-optionList">optionList()</a> = [<a href="#type-option">option()</a>]</tt></li>
<li><tt><a name="type-option">option()</a> = {max_sessions, integer()} | {response_format, <a href="#type-response_format">response_format()</a>} | {stream_chunk_size, integer()} | {max_pipeline_size, integer()} | {trace, <a href="#type-boolean">boolean()</a>} | {is_ssl, <a href="#type-boolean">boolean()</a>} | {ssl_options, [SSLOpt]} | {pool_name, atom()} | {proxy_host, string()} | {proxy_port, integer()} | {proxy_user, string()} | {proxy_password, string()} | {use_absolute_uri, <a href="#type-boolean">boolean()</a>} | {basic_auth, {<a href="#type-username">username()</a>, <a href="#type-password">password()</a>}} | {cookie, string()} | {content_length, integer()} | {content_type, string()} | {save_response_to_file, <a href="#type-srtf">srtf()</a>} | {stream_to, <a href="#type-stream_to">stream_to()</a>} | {http_vsn, {MajorVsn, MinorVsn}} | {host_header, string()} | {inactivity_timeout, integer()} | {connect_timeout, integer()} | {socket_options, Sock_opts} | {transfer_encoding, {chunked, ChunkSize}} | {headers_as_is, <a href="#type-boolean">boolean()</a>} | {give_raw_headers, <a href="#type-boolean">boolean()</a>}</tt></li>
<li><tt><a name="type-option">option()</a> = {max_sessions, integer()} | {response_format, <a href="#type-response_format">response_format()</a>} | {stream_chunk_size, integer()} | {max_pipeline_size, integer()} | {trace, <a href="#type-boolean">boolean()</a>} | {is_ssl, <a href="#type-boolean">boolean()</a>} | {ssl_options, [SSLOpt]} | {pool_name, atom()} | {proxy_host, string()} | {proxy_port, integer()} | {proxy_user, string()} | {proxy_password, string()} | {use_absolute_uri, <a href="#type-boolean">boolean()</a>} | {basic_auth, {<a href="#type-username">username()</a>, <a href="#type-password">password()</a>}} | {cookie, string()} | {content_length, integer()} | {content_type, string()} | {save_response_to_file, <a href="#type-srtf">srtf()</a>} | {stream_to, <a href="#type-stream_to">stream_to()</a>} | {http_vsn, {MajorVsn, MinorVsn}} | {host_header, string()} | {inactivity_timeout, integer()} | {connect_timeout, integer()} | {socket_options, Sock_opts} | {transfer_encoding, {chunked, ChunkSize}} | {headers_as_is, <a href="#type-boolean">boolean()</a>} | {give_raw_headers, <a href="#type-boolean">boolean()</a>} | {preserve_chunked_encoding, <a href="#type-boolean">boolean()</a>}</tt></li>
<li><tt><a name="type-stream_to">stream_to()</a> = <a href="#type-process">process()</a> | {<a href="#type-process">process()</a>, once}</tt></li>
<li><tt><a name="type-process">process()</a> = pid() | atom()</tt></li>
<li><tt><a name="type-username">username()</a> = string()</tt></li>
Expand Down Expand Up @@ -284,6 +284,11 @@ <h3 class="function"><a name="send_req-5">send_req/5</a></h3>
caller to get access to the raw status line and raw unparsed
headers. Not quite sure why someone would want this, but one of my
users asked for it, so here it is. </li>

<li> The <code>preserve_chunked_encoding</code> option enables the caller
to receive the raw data stream when the Transfer-Encoding of the server
response is Chunked.
</li>
</ul>
</p>

Expand Down Expand Up @@ -441,6 +446,6 @@ <h3 class="function"><a name="trace_on-2">trace_on/2</a></h3>
<hr>

<div class="navbar"><a name="#navbar_bottom"></a><table width="100%" border="0" cellspacing="0" cellpadding="2" summary="navigation bar"><tr><td><a href="overview-summary.html" target="overviewFrame">Overview</a></td><td><a href="http://www.erlang.org/"><img src="erlang.png" align="right" border="0" alt="erlang logo"></a></td></tr></table></div>
<p><i>Generated by EDoc, May 17 2010, 23:21:42.</i></p>
<p><i>Generated by EDoc, Sep 22 2010, 22:56:44.</i></p>
</body>
</html>
79 changes: 66 additions & 13 deletions src/ibrowse.erl
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
%%%-------------------------------------------------------------------
%% @author Chandrashekhar Mullaparthi <chandrashekhar dot mullaparthi at gmail dot com>
%% @copyright 2005-2010 Chandrashekhar Mullaparthi
%% @version 1.6.0
%% @version 2.0.0
%% @doc The ibrowse application implements an HTTP 1.1 client. This
%% module implements the API of the HTTP client. There is one named
%% process called 'ibrowse' which assists in load balancing and maintaining configuration. There is one load balancing process per unique webserver. There is
Expand Down Expand Up @@ -236,6 +236,11 @@ send_req(Url, Headers, Method, Body) ->
%% caller to get access to the raw status line and raw unparsed
%% headers. Not quite sure why someone would want this, but one of my
%% users asked for it, so here it is. </li>
%%
%% <li> The <code>preserve_chunked_encoding</code> option enables the caller
%% to receive the raw data stream when the Transfer-Encoding of the server
%% response is Chunked.
%% </li>
%% </ul>
%%
%% @spec send_req(Url::string(), Headers::headerList(), Method::method(), Body::body(), Options::optionList()) -> response()
Expand Down Expand Up @@ -266,7 +271,8 @@ send_req(Url, Headers, Method, Body) ->
%% {socket_options, Sock_opts} |
%% {transfer_encoding, {chunked, ChunkSize}} |
%% {headers_as_is, boolean()} |
%% {give_raw_headers, boolean()}
%% {give_raw_headers, boolean()} |
%% {preserve_chunked_encoding,boolean()}
%%
%% stream_to() = process() | {process(), once}
%% process() = pid() | atom()
Expand Down Expand Up @@ -302,23 +308,46 @@ send_req(Url, Headers, Method, Body, Options, Timeout) ->
Options_1 = merge_options(Host, Port, Options),
{SSLOptions, IsSSL} =
case (Protocol == https) orelse
get_value(is_ssl, Options_1, false) of
get_value(is_ssl, Options_1, false) of
false -> {[], false};
true -> {get_value(ssl_options, Options_1, []), true}
end,
case ibrowse_lb:spawn_connection(Lb_pid, Parsed_url,
try_routing_request(Lb_pid, Parsed_url,
Max_sessions,
Max_pipeline_size,
{SSLOptions, IsSSL},
Headers, Method, Body, Options_1, Timeout, 0);
Err ->
{error, {url_parsing_failed, Err}}
end.

try_routing_request(Lb_pid, Parsed_url,
Max_sessions,
Max_pipeline_size,
{SSLOptions, IsSSL},
Headers, Method, Body, Options_1, Timeout, Try_count) when Try_count < 3 ->
case ibrowse_lb:spawn_connection(Lb_pid, Parsed_url,
Max_sessions,
Max_pipeline_size,
{SSLOptions, IsSSL}) of
{ok, Conn_Pid} ->
do_send_req(Conn_Pid, Parsed_url, Headers,
Method, Body, Options_1, Timeout);
Err ->
Err
{ok, Conn_Pid} ->
case do_send_req(Conn_Pid, Parsed_url, Headers,
Method, Body, Options_1, Timeout) of
{error, sel_conn_closed} ->
io:format("Selected connection closed. Trying again...~n", []),
try_routing_request(Lb_pid, Parsed_url,
Max_sessions,
Max_pipeline_size,
{SSLOptions, IsSSL},
Headers, Method, Body, Options_1, Timeout, Try_count + 1);
Res ->
Res
end;
Err ->
{error, {url_parsing_failed, Err}}
end.
Err
end;
try_routing_request(_, _, _, _, _, _, _, _, _, _, _) ->
{error, retry_later}.

merge_options(Host, Port, Options) ->
Config_options = get_config_value({options, Host, Port}, []),
Expand All @@ -337,11 +366,27 @@ get_lb_pid(Url) ->

get_max_sessions(Host, Port, Options) ->
get_value(max_sessions, Options,
get_config_value({max_sessions, Host, Port}, ?DEF_MAX_SESSIONS)).
get_config_value({max_sessions, Host, Port},
default_max_sessions())).

get_max_pipeline_size(Host, Port, Options) ->
get_value(max_pipeline_size, Options,
get_config_value({max_pipeline_size, Host, Port}, ?DEF_MAX_PIPELINE_SIZE)).
get_config_value({max_pipeline_size, Host, Port},
default_max_pipeline_size())).

default_max_sessions() ->
safe_get_env(ibrowse, default_max_sessions, ?DEF_MAX_SESSIONS).

default_max_pipeline_size() ->
safe_get_env(ibrowse, default_max_pipeline_size, ?DEF_MAX_PIPELINE_SIZE).

safe_get_env(App, Key, Def_val) ->
case application:get_env(App, Key) of
undefined ->
Def_val;
{ok, Val} ->
Val
end.

%% @doc Deprecated. Use set_max_sessions/3 and set_max_pipeline_size/3
%% for achieving the same effect.
Expand Down Expand Up @@ -375,6 +420,10 @@ do_send_req(Conn_Pid, Parsed_url, Headers, Method, Body, Options, Timeout) ->
Options, Timeout) of
{'EXIT', {timeout, _}} ->
{error, req_timedout};
{'EXIT', {noproc, {gen_server, call, [Conn_Pid, _, _]}}} ->
{error, sel_conn_closed};
{error, connection_closed} ->
{error, sel_conn_closed};
{'EXIT', Reason} ->
{error, {'EXIT', Reason}};
{ok, St_code, Headers, Body} = Ret when is_binary(Body) ->
Expand Down Expand Up @@ -684,6 +733,10 @@ handle_call({get_lb_pid, #url{host = Host, port = Port} = Url}, _From, State) ->

handle_call(stop, _From, State) ->
do_trace("IBROWSE shutting down~n", []),
ets:foldl(fun(#lb_pid{pid = Pid}, Acc) ->
ibrowse_lb:stop(Pid),
Acc
end, [], ibrowse_lb),
{stop, normal, ok, State};

handle_call({set_config_value, Key, Val}, _From, State) ->
Expand Down
Loading

3 comments on commit d756a2b

@fdmanana
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Chandru, I see here some uncommented calls to io:format (at least 1). Shouldn't they be do_trace/2 calls?

cheers

@cmullaparthi
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well spotted. I'll fix it later tonight.

@cmullaparthi
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in ibrowse-2.0.1

Please sign in to comment.