Page 1 of 2

ADC and the new clothes

Posted: 11 Aug 2009, 13:31
by Pietry
Recently, some already known extensions for ADC are becoming more mature and it's time for their integration into the official ADC specification. Ever since the original ADC 1.0 release almost two years ago, people came with ideas and with implementations for ADC extensions. The first one to be implemented and added to the specification is PING. However, other extensions as well are in need of being more popular and official . ADC must move on and bring in new innovations in order to satisfy the current DC requirements. Perhaps in the future we will see something that will bring ADC closer to the next level filesharing.

The next version of ADC will be released in the near future. My first suggestion is that the version to be called 1.1 or something more important, because 1.0.1 suggests only a small revision of not such big importance. Another reason is that ADC versions are not released that often that we actually need a "0" in there. ( there will probably not going to be 1.0.2, 1.0.3 , 1.1.0 and so forth ) My suggestion is to version the ADC something like 1.1, 1.2 and so forth until a major change or stability with extensions will make the need of moving to 2.0.

My personal desire for the next ADC version is to include the extensions that make ADC a better protocol and bring up the things that people always appreciated about it.

Here is a list of the extensions that should be included (and wait until they are mature enough to include)
  • PING - Pinger extension, which is already approved and it will surely be in the next version
  • DFAV - Distributed favorites, the same goes for this extension as well
  • ADCS - ADC Secure , still a real specification needs to be done, but this extension is mandatory if we are to move to the next level of direct connect
  • BLOM - Bloom Filters, already with preliminary implementations, this extension also can bring the plus that can turn more attention to ADC
  • SHA1 - SHA 1 hashes, because currently ADC relies on TIGR only, even if SHA is a broken algorithm, one can still easily use it with no problems in the network
  • UCMD - User commands , because ADC clients really need this feature and it can bring a more human face to the hubs using ADC
  • REF - Hub refferal on C-C connections because it's a simple way to figure out sources for CTM attacks
  • UPQU - Upload Queue , already with implementations in clients, because users need a better way of queueing on uploads
  • TYPE - Typing notification extension, can make the protocol more IRC-like and some developers may think of using the protocol for an IRC-DC hub in order to create a more social network.
This extensions are the first that come to my mind, but at any time some other extensions may be added. My opinion is that ADC extensions should provide an official and standard mode of implementation that can be used by anyone wanting to use them, even if they are not likely to by everybody. They should all have BASE, but there is nothing bad in having more possible extensions, and let the developers choose what to implement, at their own discretion. Better to have a specification for everybody rather than having extensions spread all over the internet in all kind of ways and not have a real guideline ( like what happened with NMDC ).

When I say this I also think about the new ideas like darkKlor's MSTR admin extension, or the URI form extension that have been presented here on ADCPortal. Also, interesting ideas that are worth investigating are BigMuscle's DHT extension or the hash set for multiple files implementation, which will both add more features to ADC that will approach it to torrent-like functionality.

I do hope that the next ADC version will have this extensions included , if they are mature enough to be included and not neglige them because I think all of them are more or less important for somebody in the network who might use them, and this was the whole purpose of extensions and the extensibility of ADC, to make the protocol friendly for everybody wanting to use it.

Last but not least, I am eager to find the HI field back in the INF command because it had a practical use and it was interesting, and still implemented in many ADC software used today.

Re: ADC and the new clothes

Posted: 11 Aug 2009, 18:13
by Pretorian
I strongly disagree with your opinion that the extensions should move to BASE. Most of the extensions are not BASE material, they are just a) too difficult to require an implementation (ADCS, BLOM etc), b) too specific to require all clients or hubs to implement the feature (UPQU, TYPE etc) and c) completely in agreement with the concept of extension (SHA1 etc).

Now, obviously the desire is to have as much useful stuff in BASE. That does not mean we cannot have a large repository of extensions. That does not mean we will fall for the same thing as NMDC. The ADC project's task is to document and make developers aware what features exist and how they shall be implemented. If someone intend to implement, say, an upload queue, UPQU is the way to go.

I don't know if I've read all of the protocols suggestions mentioned here, but I'll do my best in tackling each item.
  • While I'm sure most, if not all, major hub list pingers and hub software will implement PING due to its sheer usefulness, the extension could stay as is. Problems; a) a requirement that hubs respond to a PING request, which they might not desire, b) hubs may lie in their data, c) it's usefulness is very limited as there will be only a handful of pingers. (I'm not bashing the extension; I think its very existance display that there's a common way of pinger-hub communication.)
  • DFAV should also not be a requirement in BASE for simply the fact that users may not want to disclose their information.
  • While ADCS and BLOM are definitely some things that should be tackled, their complexity are too great. ADCS is an extension in most other protocols (HTTP, FTP etc) and should remain as such in ADC.
  • Hash algorithms should not be in BASE. Also, there's little point in actually adding SHA-1 specifically since a) TTH seem to be robust and b) next generation of SHA is on its way, but you're free to do so.
  • While there's not much immediate problems with UCMD, no one has implemented it (as far as I know). (Or DC++ has?)
  • I have no read what "REF" is, but a RF field in the C-C INF should suffice, no? There's no need to invent another command or anything... (If 'yes' to my question, then definitely as part of BASE.)
  • UPQU seem to imply that users always want to see queue uploads; I, like presumably many people, could not care less about who wants to download and what they want. The teller at the local super marker probably have no interest in knowing what the people in their queue are going to purchase. (Yeah, crappy analogy, but it's all I could think of.)
  • I'm not sure how TYPE is similar to IRC. I wasn't aware that IRC require clients to send, in real time, that someone is writing something in their client. (MSN, ICQ etc does, though.) (Also, IRC is _huge_.) (And, well, it's bloat...)
  • I'm sure most are aware that the MSTR extension(s) is _very_ large. Assuming that all clients and hubs shall implement these is just naïve, I think. (I like the concepts behind the extensions, but I think they need to be broken down more to be a) included in BASE and b) more easily understood.)
  • The URI form is, in my mind, very complex, even if I intended it to be simple. However, the idea was that no one else should have to re-invent a specification for a longer version than the BASE URI.
  • Having not read the DHT proposal, I'm fond of the very idea (as an extension and in BASE).
  • The hash set should be perhaps be something completely different and more general; meta-data exchange. I'm also contemplating whether what I wrote was/is the best solution.
  • I don't know where you see that HI is in the INF... It's not in the BASE or extensions document. (I have no opinion whether it should be in BASE or not. Its usefulness is very limited, at best.)
(Please, don't feel like I'm picking on your extension; I simply think that most of the extensions shall remain as they should be treated as optional [duh!].)

Re: ADC and the new clothes

Posted: 11 Aug 2009, 18:38
by Toast
REF was created to trace CTM connections if a malicious hub was to be used in an CTM DDoS Attack

http://www.adcportal.com/wiki/index.php/REF

Arne approved and implemented it into DC++0.75 its in the changelog of DC++ and the patch is on LP :)
from changelog wrote:Notes:
0.750 marks the moment where the migration to MinGW is mature enough to call DC++ a full MinGW compliant application. This release improves the program functionality and how it works on your system. Major enhancements include : openssl fix for older libraries, fixed major known crashes, updated help, minor GUI features, new interface for settings page, and the REF extension for NMDC and ADC that allow peers attacked by the CTM exploit(willingly or unwillingly) to find out the corrupted hub that is used for attacking.
Have fun using DC++ !

Re: ADC and the new clothes

Posted: 11 Aug 2009, 18:43
by Crise
I agree with Pret here.... but the thing which I do disagree on is UPQU. While I do agree that not all users may want this and the uploading user actually does most likely not care about it (when he isn't downloading himself).

But it should be BASE requirement imo. because of equality... in what other P2P network do you see upload queuing as something optional. Since UPQU affects the way slots are handed out, it should either be done by everyone or not at all (so that every user X can get slots in equal fashion no matter if he tries to download from Y or Z).

Also an important point regarding UPQU, if it would become a requirement then there would be no benefit in creating a client that does "slot hammering" (since with UPQU, no matter how much you hammer your position does not get affected).

And to make it clear regarding Upload Queue, it's not for the uploading user at all really, nor for visual stuff... but for the downloaders to make everyone stand on equal ground (see slot hammering above f.ex.).

UCM0 has been done in DC++ afaik (don't know if it is called UCMD by now).

Re: ADC and the new clothes

Posted: 11 Aug 2009, 20:14
by Pretorian
REF is not in the spec, see http://adc.svn.sourceforge.net/viewvc/adc/trunk/.

I'll add it, since it seems to be the general concensus that it should be part of the spec. I'm not sure where to put it, though... Also, in my mind, there'd be a special "error code" field so implementations wouldn't display this to the user...

Re: ADC and the new clothes

Posted: 12 Aug 2009, 07:33
by Pietry
(Please, don't feel like I'm picking on your extension; I simply think that most of the extensions shall remain as they should be treated as optional [duh!].)
Pret you misinterpreted my affirmations. I did not say add all this to BASE. The sole modification that I propose to BASE is the HI field in INF (which was there in previous ADC drafts but removed ) . The rest are just extensions that need to be added to official supported extensions, not to BASE. BASE must remain as plain and as simple as it is.
I say this: let the developer decide, but offer him a real specification, that's all. Nobody is forcing anyone to implement any of this extensions or use them. All I want is for the extensions to be available for everyone so that everyone has the chance to see them and make the same implementation.

Re: ADC and the new clothes

Posted: 12 Aug 2009, 08:15
by Pietry
Pretorian wrote:While I'm sure most, if not all, major hub list pingers and hub software will implement PING due to its sheer usefulness, the extension could stay as is. Problems; a) a requirement that hubs respond to a PING request, which they might not desire, b) hubs may lie in their data, c) it's usefulness is very limited as there will be only a handful of pingers. (I'm not bashing the extension; I think its very existance display that there's a common way of pinger-hub communication.)
PING is the only way to get some current hub settings that are mandatory for a hublist ( I think about minimum share, maximum user count and so forth ).
Pretorian wrote: DFAV should also not be a requirement in BASE for simply the fact that users may not want to disclose their information.
It's not supposed to be part of BASE.
Pretorian wrote:While ADCS and BLOM are definitely some things that should be tackled, their complexity are too great. ADCS is an extension in most other protocols (HTTP, FTP etc) and should remain as such in ADC.
That was my opinion as well in the beginning.
Pretorian wrote:Hash algorithms should not be in BASE. Also, there's little point in actually adding SHA-1 specifically since a) TTH seem to be robust and b) next generation of SHA is on its way, but you're free to do so.
I agree, but I feel the need to let the developer decide, if he wants to implement it.
Pretorian wrote: While there's not much immediate problems with UCMD, no one has implemented it (as far as I know). (Or DC++ has?)
DC++ has preliminary implementation, also eHub IIRC.
Pretorian wrote: I have no read what "REF" is, but a RF field in the C-C INF should suffice, no? There's no need to invent another command or anything... (If 'yes' to my question, then definitely as part of BASE.)
That's exactly what REF is, and sure, it's an extension it doesn't have to be part of BASE.
Pretorian wrote:UPQU seem to imply that users always want to see queue uploads; I, like presumably many people, could not care less about who wants to download and what they want. The teller at the local super marker probably have no interest in knowing what the people in their queue are going to purchase. (Yeah, crappy analogy, but it's all I could think of.)
UPQU just gives everybody a fair chance in getting a slot .
Pretorian wrote:I'm not sure how TYPE is similar to IRC. I wasn't aware that IRC require clients to send, in real time, that someone is writing something in their client. (MSN, ICQ etc does, though.) (Also, IRC is _huge_.) (And, well, it's bloat...)
Bloat or not, the extension is supposed to give the chance to someone wanting such a feature
Pretorian wrote: I'm sure most are aware that the MSTR extension(s) is _very_ large. Assuming that all clients and hubs shall implement these is just naïve, I think. (I like the concepts behind the extensions, but I think they need to be broken down more to be a) included in BASE and b) more easily understood.)
They need to be broken down but not included in BASE, just extensions
Pretorian wrote: The hash set should be perhaps be something completely different and more general; meta-data exchange. I'm also contemplating whether what I wrote was/is the best solution.
Perhaps something like SDP ?

Re: ADC and the new clothes

Posted: 12 Aug 2009, 09:39
by Dj_Offset
For ADC I'd like two things.

1) It needs to be versioned, so that if we change the protocol we can still provide a fallback approach to old clients.

2) I would actually like to remove things from BASE. Think outside the box, i.e no filesharing/searches, no chat.

I presented both these comments a couple of years ago when ADC 0.12 was fresh and they went pretty much uncommented, except for the
really boring details which were actually fixed in the spec.

Link: http://www.extatic.org/adc/adc_0.12_discussion.txt

Re: ADC and the new clothes

Posted: 12 Aug 2009, 10:02
by cologic
From the DC Development public hub:
<cologic> "SHA1 - SHA 1 hashes, because currently ADC relies on TIGR only, even if SHA is a broken algorithm, one can still easily use it with no problems in the network" ugh no
<cologic> SHA2/SHA256? yeah sure, I guess. But not SHA1 of all things.
<cologic> (Neutral/positive on rest.)
<Toast> im pretty sure that cyb isnt gonna finish it
<Toast> so i could remove it since it seems to be an eyesore
<cologic> I like the idea of moving to another hash eventually, as Tiger is a round or two away from being broken - but wait for SHA3 for that ( http://csrc.nist.gov/groups/ST/hash/sha-3/ )
<cologic> So long as Tiger can remain workable until the "Second SHA-3 Candidate Conference is being planned for August 23-24, 2010, after Crypto 2010", and somewhat later, it should be fine.
<darkKlor> I think SHA1 is workable in the sense that who will bother to actively exploit it? But no I don't think it is necessary... wait for SHA3. Torrents use SHA1.. is the format being urgently rewritten? I doubt it..
<Dj_Offset> i submitted a comment on that post too
<cologic> darkKlor: Tiger is workable in the same sense.
<Dj_Offset> even more so, I would say
<cologic> And SHA1 is actually in worse condition than Tiger.
<cologic> So since there's a real cost engineeringwise of supporting multiple hashes...
<Dj_Offset> actually, if SHA1 is *truely* broken, most e-commerce must stop... but that isn't the case... we are phasing out MD2 and MD5 from SSL certificates these days
<Dj_Offset> SHA1 will live for 5+ years at *least*
<cologic> Yeah - the SHA1 breaks are still fairly special-purpose and limited to people with unusually large computational resources. But it's trivially parallelizable and the breaks will only get better.
<darkKlor> re: versioning. yeah I wondered when Pietry said lets make v1.1 of ADC... how will that be announced by hubs and clients? BAS0, BASE, BAS2 doesn't give much versioning versatility
<cologic> (and constructions such as HMAC mitigate much of the harm)
<cologic> Dj_Offset: I'd suggest as an alternate explanation for the lack of panic in the ecommerce industry is that there alternate attack methods are still much easier and will be for a while.
<Dj_Offset> cologic: yes, ...and SHA-3 is speeding up, but is still years away from a SSL point of view.
<Dj_Offset> btw, in ADC I'd say the hash used for password challenge response is not critical, but the one used for file hashes
<Dj_Offset> which means it becomes easier for **AA and others to produce fake files
<Dj_Offset> ...which is slightly mitigated by the merkle tree stuff, as long as a client actually transfers the leaf nodes first
<Dj_Offset> (from a trusted source, that is...)
<cologic> Dj_Offset: yup. I hope everyone on eMule/eDonkey has abandoned its initial MD4 hashes for the SHA1-based ICH hashes long ago for that reason...
<Dj_Offset> yes, but producing fake files is still remarkably easy, because *who* validates the hash itself?
<cologic> Sure, but that's an out-of-band trust issue.
<cologic> On the other of MITMs
<cologic> order of...
<Dj_Offset> ...and most importantly filenames
<Dj_Offset> so, all in all, I'd say we can use Tiger for years to come
<cologic> ... and as such safely wait for SHA3 rather than use an already-broken SHA1 or a soon-to-be-obsolete SHA2.
<Dj_Offset> right

Re: ADC and the new clothes

Posted: 12 Aug 2009, 17:07
by Pretorian
Pietry: Sorry, I misunderstood you then, especially since you said "some already known extensions for ADC are becoming more mature and it's time for their integration into the official ADC specification", so I assumed you meant BASE. Fair enough, then. :)