Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enrich ConnectionPoolListener interface to listen more connection events #5580

Open
wants to merge 24 commits into
base: main
Choose a base branch
from

Conversation

festinalente91
Copy link

@festinalente91 festinalente91 commented Apr 8, 2024

Motivation:

ConnectionPoolListener provides a limited way to listen to connection events. (open, close)
We could provide a more sophisticated event interface like the one below.

  • Connection pending
  • Connection failed
  • Connection active
  • Connection idle
                                                                                           
                                    +------------------+                                   
        +--------------+            | +--------------+ |            +--------------+       
        |              |            | |              | |            |              |       
        |    Pending   |----------->| |    Opened    | |----------> |    Closed    |       
        |              |            | |              | |            |              |       
        +--------------+            | +--------------+ |            +--------------+       
                |                   |                  |                                   
                |                   |      Active      |                                   
                |                   |         ^        |                                   
                |                   |         |        |                                   
                |                   |         |        |                                   
                v                   |         |        |                                   
        +--------------+            |         |        |                                   
        |              |            |         v        |                                   
        |    Failed    |            |       Idle       |                                   
        |              |            |                  |                                   
        +--------------+            +------------------+                                   

Furthermore, as @ikhoon mentioned here
we can also expand this connection event interface to be used on the server side.

Modifications:

  • (Deprecated) com.linecorp.armeria.client.ConnectionPoolListener class is deprecated.
    • Use com.linecorp.armeria.client.ClientConnectionEventListener instead.

Result:

@CLAassistant
Copy link

CLAassistant commented Apr 8, 2024

CLA assistant check
All committers have signed the CLA.

@festinalente91 festinalente91 force-pushed the feature/enrich-connection-pool-listener-event-interface branch from b52f5a3 to 2851ee6 Compare April 8, 2024 08:41
@ikhoon ikhoon added new feature sprint Issues for OSS Sprint participants labels Apr 8, 2024
@festinalente91 festinalente91 force-pushed the feature/enrich-connection-pool-listener-event-interface branch 2 times, most recently from 9f7edb8 to 1da16f2 Compare April 17, 2024 01:12
@@ -181,6 +189,26 @@ public final void onReadOrWrite() {
lastConnectionIdleTime = System.nanoTime();
}

if (!isServer && !isActive) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be useful if we also support ConnectionPoolListener for servers. Although we are focusing on the client-side implementation, the public API design should take extensibility into consideration.

I propose to add a new ConnectionPoolListener to c.l.a.common package. The new methods will be added to the new ConnectionPoolListener which extends the old ConnectionPoolListener for compatibility. For example:

public interface StreamDecoder extends com.linecorp.armeria.client.encoding.StreamDecoder {

Related issue: #3722

@festinalente91 festinalente91 force-pushed the feature/enrich-connection-pool-listener-event-interface branch from 416f42b to 4be97ef Compare April 21, 2024 10:38
Copy link

codecov bot commented Apr 21, 2024

Codecov Report

Attention: Patch coverage is 78.20513% with 17 lines in your changes missing coverage. Please review.

Project coverage is 74.03%. Comparing base (e08527d) to head (4be97ef).
Report is 6 commits behind head on main.

Current head 4be97ef differs from pull request most recent head b8873e1

Please upload reports for the commit b8873e1 to get more accurate results.

Files Patch % Lines
...eria/internal/common/AbstractKeepAliveHandler.java 66.66% 8 Missing and 2 partials ⚠️
.../armeria/client/ConnectionPoolListenerWrapper.java 0.00% 4 Missing ⚠️
...a/com/linecorp/armeria/client/HttpChannelPool.java 50.00% 1 Missing ⚠️
...inecorp/armeria/common/ConnectionPoolListener.java 0.00% 1 Missing ⚠️
...linecorp/armeria/common/ConnectionPoolMetrics.java 96.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #5580      +/-   ##
============================================
- Coverage     74.75%   74.03%   -0.73%     
+ Complexity    21409    21197     -212     
============================================
  Files          1877     1844      -33     
  Lines         79126    78438     -688     
  Branches      10201    10010     -191     
============================================
- Hits          59152    58069    -1083     
- Misses        15179    15668     +489     
+ Partials       4795     4701      -94     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@festinalente91 festinalente91 force-pushed the feature/enrich-connection-pool-listener-event-interface branch from 918d3ac to 6250594 Compare April 28, 2024 03:42
@festinalente91 festinalente91 force-pushed the feature/enrich-connection-pool-listener-event-interface branch 5 times, most recently from edcff6b to 5a4530a Compare May 7, 2024 01:52
@festinalente91 festinalente91 requested a review from ikhoon May 7, 2024 04:46
@festinalente91 festinalente91 marked this pull request as ready for review May 7, 2024 04:50
Copy link
Member

@trustin trustin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really good. Thanks for your high quality contribution, @festinalente91!

/**
* Listens to the client connection pool events.
*/
public interface ConnectionPoolListener extends com.linecorp.armeria.client.ConnectionPoolListener {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On second thought, ConnectionEventListener would be a better name for the new API than ConnectionPoolListener. Because:

  • If we use the same name, setter methods can be easily overloaded but it is difficult to name the new API while deprecating the old one.
  • pool is not a good word for the server side. Servers accept incoming connections but do not manage pooled connections for reuse. Therefore, a more general term, event would make more sense.
Suggested change
public interface ConnectionPoolListener extends com.linecorp.armeria.client.ConnectionPoolListener {
public interface ConnectionEventListener {

Should we deprecate ClientFactoryBuilder.connectionPoolListener() and related methods in favor of ClientFactoryBuilder.connectionEventListener()?
For backward compatibility, we need to convert ConnectionPoolListener to ConnectionEventListener internally.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the name. 😊

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ikhoon

Should we deprecate ClientFactoryBuilder.connectionPoolListener() and related methods in favor of ClientFactoryBuilder.connectionEventListener()?

Yes, I think so.

For backward compatibility, we need to convert ConnectionPoolListener to ConnectionEventListener internally.

Okay. I'll convert it.👍

Copy link
Author

@festinalente91 festinalente91 May 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I converted the deprecated ConnectionPoolListener into ClientConnectionEventListener in the ClientFactoryBuilder. if there's already CONNECTION_EVENT_LISTENER, then it just ignores the ConnectionPoolListener.

// ClientConnectionEventListener.class
    static final ClientConnectionEventListenerAdapter NOOP = new ClientConnectionEventListenerAdapter();

    /**
     * Convert the {@link ConnectionPoolListener} into a {@link ClientConnectionEventListener}.
     */
    static ClientConnectionEventListener of(ConnectionPoolListener connectionPoolListener) {
        return new ClientConnectionEventListenerAdapter() {
            @Override
            public void connectionOpened(@Nullable SessionProtocol desiredProtocol,
                                         SessionProtocol protocol,
                                         InetSocketAddress remoteAddress,
                                         InetSocketAddress localAddress,
                                         AttributeMap attrs) throws Exception {
                connectionPoolListener.connectionOpen(protocol,
                                                      remoteAddress,
                                                      localAddress,
                                                      attrs);
            }

            @Override
            public void connectionClosed(SessionProtocol protocol,
                                         InetSocketAddress remoteAddress,
                                         InetSocketAddress localAddress,
                                         boolean isIdle,
                                         AttributeMap attrs) throws Exception {
                connectionPoolListener.connectionClosed(protocol,
                                                        remoteAddress,
                                                        localAddress,
                                                        attrs);
            }
        };
    }
// ClientFactoryBuilder.class

    private ClientFactoryOptions buildOptions() {
       ...
        if (options.containsKey(ClientFactoryOptions.CONNECTION_POOL_LISTENER) &&
            !options.containsKey(ClientFactoryOptions.CONNECTION_EVENT_LISTENER)) {
            final ConnectionPoolListener connectionPoolListener =
                    (ConnectionPoolListener) options.get(ClientFactoryOptions.CONNECTION_POOL_LISTENER).value();

            options.put(ClientFactoryOptions.CONNECTION_EVENT_LISTENER,
                        ClientFactoryOptions.CONNECTION_EVENT_LISTENER
                                .newValue(ClientConnectionEventListenerAdapter.of(connectionPoolListener)));
        }

@festinalente91 festinalente91 force-pushed the feature/enrich-connection-pool-listener-event-interface branch from 5a4530a to 89e0257 Compare May 12, 2024 03:34
Copy link
Member

@minwoox minwoox left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Basically, looks good. 👍

final ConnectionEventKey connectionEventKey = connectionEventKey(channel);

try {
connectionEventListener.connectionOpened(connectionEventKey, channel);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If NoopKeepAliveHandler is used, this event isn't called anymore.

keepAliveHandler = new NoopKeepAliveHandler();

Is there any reason that you've moved this logic from HttpChannelPool?

Copy link
Author

@festinalente91 festinalente91 May 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to move open and close event trigger logic into the AbstractKeepAliveHandler to reuse this on the server side. However, as you said, I had to make extra changes to do that. Any feedback for this? @minwoox @ikhoon

  • Added connection opened and closed event trigger logic into the NoopKeepAliveHandler
    • To share the open and close event trigger logic on server side.
  • Handled HTTP Upgrade cases with context-checking(HttpProtocolUpgradeHandler)
    • When the protocol is upgraded, AbstractKeepAliveHandler is called twice.
    • So, I had to ignore the first one which was related to the protocol upgrade process.
  • (Still struggling to handle HTTP2 Max connection setting test case.
    • void exceededMaxStreamsPropagatesFailureCorrectly() throws Exception {)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to move open and close event trigger logic into the AbstractKeepAliveHandler to reuse this on the server side

Since the connection mechanisms differ between the server-side and client-side, I don't think we have to put the logic into the same class.
Instead, we could consider incorporating the logic for the server-side into ConnectionLimitingHandler, which is already used for managing connections. You can find the ConnectionLimitingHandler implementation here: Link

@festinalente91 festinalente91 force-pushed the feature/enrich-connection-pool-listener-event-interface branch 2 times, most recently from e99fa5d to 6e4e7e5 Compare May 14, 2024 13:25
@festinalente91 festinalente91 force-pushed the feature/enrich-connection-pool-listener-event-interface branch from aa47532 to 7564086 Compare June 2, 2024 15:29
@minwoox minwoox added this to the 1.30.0 milestone Jun 3, 2024
Copy link
Contributor

@ikhoon ikhoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the late review. I'll take a review in detail tomorrow.

/**
* A key to identify a connection event.
*/
public final class ConnectionEventKey {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move to the internal package? I guess there is no API in which this class is exposed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, we could also move to the client package and make it package private.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jrhee17 -nim
Since this ConnectionEventKey class might be used in the server package, how about putting this under the internal.common package?

final ConnectionEventKey connectionEventKey = connectionEventKey(ctx.channel());

try {
connectionEventListener.connectionIdle(protocol,
Copy link
Contributor

@ikhoon ikhoon Jun 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to gather idle and keep-alive related logic into KeepAliveHandler. It is doable to track idle connections in AbstractHttpResponseDecoder but the logic could be fragmented.

We already call KeepAliveHandler when a new request is made.


How about adding keepAliveHandler().decreaseNumRequests() and just calling it in AbstractHttpResponseDecoder.removeResponse() instead?

Copy link
Contributor

@jrhee17 jrhee17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, left some minor comments 👍

public ClientFactoryBuilder connectionPoolListener(
ConnectionPoolListener connectionPoolListener) {
option(ClientFactoryOptions.CONNECTION_POOL_LISTENER,
requireNonNull(connectionPoolListener, "connectionPoolListener"));
return this;
}

/**
* Sets the listener which is notified on a connection event.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also update the javadocs to let users know that connectionEventListener won't have an effect if connectionPoolListener is set?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay! I would leave the comment that connectionPoolListener won't have an effect if connectionEventListener is set.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I slightly prefer raising an IllegalStateException when both ConnectionPoolListener and ClientConnectionEventListener are set.
We may store a specified ClientConnectionEventListener in a member field and set to options in buildOptions().

public ClientFactoryBuilder connectionPoolListener(
    ConnectionPoolListener connectionPoolListener) {
    checkState(connectionEventListener == null, "connectionPoolListener() and
        connectionEventListener() are mutually exclusive")
    this.connectionPoolListener = connectionPoolListener;
}

...

private ClientFactoryOptions buildOptions() {
    if (connectionPoolListener != null) {
       // Convert connectionPoolListener to connectionEventListener.
    }
}

/**
* A key to identify a connection event.
*/
public final class ConnectionEventKey {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, we could also move to the client package and make it package private.

/**
* Returns the desired protocol of the connection. This field is only used for client-side connection.
*/
@Nullable
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit; This doesn't have to be nullable if the ctor requires a desiredProtocol. We can also clean up related code.

Suggested change
@Nullable

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I designed the ConnectionEventKey class (and ConnectionEventListener interface) for both the server and the client. Since the desired protocol only exists on the client side, this field has to be nullable for the server side.
Or perhaps we could extend this class to be more client-specific, but I'm getting worried that would make the code unnecessarily hierarchical and hard to read.

Copy link
Contributor

@jrhee17 jrhee17 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some more comments 👍

Comment on lines 439 to 440
// When the transport type(-Dcom.linecorp.armeria.transportType) is NIO,
// sometimes local address is not available yet.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is because the channel isn't bound yet at this point. Is there a reason we can't just remove L434-L445?

@@ -110,6 +112,7 @@ final class HttpChannelPool implements AsyncCloseable {
SessionProtocol.H2, SessionProtocol.H2C));
pendingAcquisitions = newEnumMap(httpAndHttpsValues());
allChannels = new IdentityHashMap<>();
connectionEventStates = new IdentityHashMap<>();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question) What is the reason we need this map?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the sessionPromise failed, I could not get the channel because the future.getNow() returns null.
Also, I couldn't pass the ConnectionEventState to the notifyConnectionFailed method, because the ConnectionEventState is created in the callback.

/**
* Creates a new instance.
*/
public ConnectionEventState(InetSocketAddress remoteAddress,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like maintaining the state is doing us more harm than good at this point. What do you think of just removing ConnectionEventState and using ChannelUtil#localAddress, ChannelUtil#remoteAddress, HttpChannelPool#getProtocolIfHealthy when notifying events instead?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that we'll also have to do bookkeeping on NoopKeepAliveHandler if we're committing to maintaining state in KeepAliveHandler like @ikhoon suggested.

@@ -64,4 +64,10 @@ public boolean needsDisconnection() {

@Override
public void increaseNumRequests() {}

@Override
public void tryNotifyConnectionActive() {}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think of just adding a decreaseNumRequests at the location suggested by @ikhoon at this comment?
#5580 (comment)

/**
* Convert the {@link ConnectionPoolListener} into a {@link ClientConnectionEventListener}.
*/
static ClientConnectionEventListener of(ConnectionPoolListener connectionPoolListener) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we place this method in ClientConnectionEventListener?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better to move this to the ClientFactoryBuilder. 👍

public ClientFactoryBuilder connectionPoolListener(
ConnectionPoolListener connectionPoolListener) {
option(ClientFactoryOptions.CONNECTION_POOL_LISTENER,
requireNonNull(connectionPoolListener, "connectionPoolListener"));
return this;
}

/**
* Sets the listener which is notified on a connection event.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I slightly prefer raising an IllegalStateException when both ConnectionPoolListener and ClientConnectionEventListener are set.
We may store a specified ClientConnectionEventListener in a member field and set to options in buildOptions().

public ClientFactoryBuilder connectionPoolListener(
    ConnectionPoolListener connectionPoolListener) {
    checkState(connectionEventListener == null, "connectionPoolListener() and
        connectionEventListener() are mutually exclusive")
    this.connectionPoolListener = connectionPoolListener;
}

...

private ClientFactoryOptions buildOptions() {
    if (connectionPoolListener != null) {
       // Convert connectionPoolListener to connectionEventListener.
    }
}

@@ -92,16 +93,17 @@ final class HttpChannelPool implements AsyncCloseable {
// Fields for pooling connections:
private final Map<PoolKey, Deque<PooledChannel>>[] pool;
private final Map<PoolKey, ChannelAcquisitionFuture>[] pendingAcquisitions;
private final Map<Future<Channel>, ConnectionEventState> connectionEventStates;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we pass ConnectionEventState as an argument instead? I'm unsure Future<Channel> is a good key and we may implement without this.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that the Future is not a good key and passing it as an argument is much cleaner.
However, since the ConnectionEventState is created inside the sessionPromise callback, there is no way to pass it to the notifyConnect method.

@@ -407,14 +410,39 @@ void connect(SocketAddress remoteAddress, SessionProtocol desiredProtocol,
// to HttpSessionHandler.
connectionPromise.addListener((ChannelFuture connectFuture) -> {
if (connectFuture.isSuccess()) {
// When the transport type(-Dcom.linecorp.armeria.transportType) is NIO,
// sometimes local address is not available yet.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@@ -407,14 +410,39 @@ void connect(SocketAddress remoteAddress, SessionProtocol desiredProtocol,
// to HttpSessionHandler.
connectionPromise.addListener((ChannelFuture connectFuture) -> {
if (connectFuture.isSuccess()) {
// When the transport type(-Dcom.linecorp.armeria.transportType) is NIO,
// sometimes local address is not available yet.
if (ChannelUtil.connectionEventState(channel) == null) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a connection is just created, is this condition always true?

Comment on lines 434 to 445

final InetSocketAddress remoteInetAddress = poolKey.endpoint.toSocketAddress(-1);
final InetSocketAddress localAddress = ChannelUtil.localAddress(channel);

if (localAddress == null) {
// When the transport type(-Dcom.linecorp.armeria.transportType) is NIO,
// sometimes local address is not available yet.
return;
}

notifyConnectionPending(desiredProtocol, remoteInetAddress, localAddress,
channel, sessionPromise);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's remove and keep the approach above.

@ikhoon
Copy link
Contributor

ikhoon commented Jun 27, 2024

This became a big PR with more than 2000 lines. As a first-time contributor, you are not familiar with Armeria's code internals and conventions. It is also not easy for us to review over 2000 lines.

Let me add a clean-up commit and build together. 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new feature sprint Issues for OSS Sprint participants
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Enrich ConnectionPoolListener interface to listen to more connection events
6 participants