Blom causes big latencies

Here is the sub forum used for talking about ideas, implementations and suggestions or typical guidelines.

Further info on extension or the protocol is found at our Wiki
Locked
klondike
Member
Posts: 73
Joined: 14 Nov 2010, 13:06

Blom causes big latencies

Post by klondike » 29 Nov 2012, 16:01

I have discovered that BLOM has been the responsible of a big set of related issues involving big latencies, from the classical "It takes to long to authenticate" to "I get big latencies on my messages when hashing". This seems to be caused by the fact the filter is sent using the same channel as the rest of the data to the hub which causes a big bottleneck when said filter is big if a big number of files is available.

I'd like that we discuss and find a solution to this issue if you have time for it. Some coming out of the back of my head are:
* Set some escape sequence during file transfer that can be used to pause the transfer so other commands can be sent over the same channel.
* Send over packets the data transfers so other information can be interleaved (there are multiple ways to do it)
* Related to the previous two: Have switchable modes (suggested by iceman)
* Try to use a second channel for file transfers (either a second connection established with a token or, when using SSL a second channel).

poy
Member
Posts: 78
Joined: 26 Nov 2008, 17:04

Re: Blom causes big latencies

Post by poy » 29 Nov 2012, 17:12

perhaps clients could simply split the data into smaller chunks, sent with multiple SND messages?

looking at the ADCH++ source, i'm guessing it would refuse partial tables with the "Invalid number of bytes" message; but it shouldn't be very difficult to amend that.

poy
Member
Posts: 78
Joined: 26 Nov 2008, 17:04

Re: Blom causes big latencies

Post by poy » 29 Nov 2012, 17:15

poy wrote:perhaps clients could simply split the data into smaller chunks, sent with multiple SND messages?

looking at the ADCH++ source, i'm guessing it would refuse partial tables with the "Invalid number of bytes" message; but it shouldn't be very difficult to amend that.
an alternative that may be more in line with common ADC conventions would be for the hub to do that splitting, by requesting partial data in the GET messages that it sends.

poy
Member
Posts: 78
Joined: 26 Nov 2008, 17:04

Re: Blom causes big latencies

Post by poy » 29 Nov 2012, 17:20

or hubs could just put an upper bound on the m parameter.

that's actually suggested in the algo description: "Also, with m grows the required bandwidth to transfer the bloom filter, so the hub may wish to cap m. In that case, it should still choose k according to m / n * ln 2, but send an h as big as possible to alleviate biasing problems."
all that will happen with smaller ms is that the probability of false positives will increase (less searches filtered).

kathi
Newbie
Posts: 1
Joined: 21 Nov 2014, 11:40

Re: Blom causes big latencies

Post by kathi » 21 Nov 2014, 11:43

I've currently switched to the other extreme: now using the (cid, hub) pair for everything. That at least guarantees that nothing will go wrong in that aspect. I can always make better use of CIDs and possibly provide fallbacks later on.
kathi

Locked